First of all, a little terminology. Your best performing campaign or salespiece is your Champion, or your Control. Once you know your numbers (of responses, conversions, etc.), the objective in testing is to find a way to beat your control with a Challenger, or a Test. The winner becomes the new Control, and the cycle starts over, constantly seeking improvement.
When you’re testing, make only one change at a time, if at all possible. You can’t always determine the date a letter will be received, but you can test different drop dates. You can’t always ensure that an ad has the same position on a page, what or who else may be advertising at the same time, the exact time of your spot, or any of a hundred other variables, but you can do a “A-B Split,” with ½ of the run carrying your Control ad, the other ½ the Challenger ad. So pick variables you have some control over and go with those.
I feel the very best medium to test in is direct mail. This is because mailing lists are so defined, so segmented, that you can break down your market almost infinitely. Also, you can send out multiple tests to the same demographic universe, and get solid numbers that tell you what works and what doesn’t (or what works better).
You can make sure that you only test one variable at a time. You can change only your headline, or your font, or your envelope, or your price or whatever else you have control over. Do NOT try to change the headline, add color, change the offer and the fast-response bonus all in the same test. How will you know what change made the difference? Too many simultaneous changes will yield results that mean nothing. Junk.
Change only one variable at a time. This way, you can look at your results knowing that the only change you made was to x, so having the new x in the campaign made a positive (or negative) difference in response.
