The term universe describes the people-group or market list we are marketing into. Your control piece (or package or show) is the previous version that has had the greatest response so far. Always try to test your marketing to the same people, or demographic universe as possible. The only exception to this is when you are testing the same Control piece into a different universe (to see if you get similar results on new list, to your previously measured results).
Using whatever control piece you have, first mail to a small segment of your chosen universe, to satisfy yourself that you have a productive list. I usually like to have at least 1000-2000 names, to have some degree of statistical accuracy. This is not enough for certainty, but only enough to tell me if the list is responsive enough to test more pieces. Try to make sure your chosen list has enough names for a potential rollout… several hundred thousand is always nice! Don’t just throw your test piece to any available list.
Next, using the new piece with your subtle variable change, test into the same universe. I like to always test 10 – 20% of any mailing using a new Challenger to my Control (so 80% of a particular mailing will receive the Control piece). This is also called an “A – B Test,” perhaps with your Control the “A” letter, the other letter your Challenger, “B.” Measure your percentage of results against your Control piece. This is the Validation Test, validating the original small test with the same results (or invalidating it with different results).
While there are many variables you may not be aware of (like ones we’ve already discussed), the two most important are your list (universe) and your control. Once you’ve established your control results to a given list, make your changes to your control piece to test against it, testing the response of the same list (not necessarily the same names on the list). Ultimately, you objective is to have a new Champion that is superior to your original Control.
When you test into a new universe, test the first control against the new, yielding comparable results of one list to the other for both pieces. This gives you two comparisons with the original list, and will help you to determine how the second list will perform compared to the first list. At the same time, you are retesting the effectiveness of your control pieces (by verifying the new results for each vs. the old results, against a new universe).
