Most A/B tests on small lists prove nothing. They still produce a winner, a screenshot, and a lesson that gets applied to the next six campaigns, which is what makes them expensive rather than merely useless.
The failure is almost always the same. A list of 800 gets split in half, variant B opens 3 points higher, B is declared the winner, and a rule enters the team's folklore. At 400 per side you cannot reliably see anything smaller than about a 10-point gap, so a 3-point difference is comfortably inside the range you would get by splitting the list at random and sending the same email to both halves.
One variable
Split on the subject or split on the body. Never both in the same test.
This sounds obvious and is broken constantly, usually because two versions of an email already existed and testing them against each other felt free. If B has a different subject and a rewritten opener and B wins, you have learned that B is better. You have learned nothing you can reuse.
The other half of the rule: the metric has to match the variable.
- Subject test → read on open rate. A subject line cannot influence anything downstream of the open.
- Body test → read on clicks or replies. Both variants had the same subject, so their open rates should be identical, and if they are not you have a randomisation bug rather than a result.
How big the slice needs to be
Around 1,500 recipients per variant if you want to trust a 5-point difference in open rate. That is not a rule of thumb, it is what a two-proportion power calculation gives at a 40% baseline open rate, 80% power and 95% confidence, and it is the number most "just split your list" advice is quietly ignoring.
| Per variant | Smallest difference you can trust | Verdict |
|---|---|---|
| 250 | ~12 points | Only a landslide shows up |
| 500 | ~9 points | Big swings only |
| 1,000 | ~6 points | Borderline, and usually read as if it were better |
| 1,500 | ~5 points | The practical floor for a subject test |
| 5,000 | ~3 points | Fine-grained testing becomes possible |
Those are minimum detectable differences, not guarantees. A gap smaller than the row you are sending at is a tie, however much it looks like a result.
If your whole list is 1,200 people, you cannot run a meaningful subject test on it in one campaign. You can still test, but the honest way is to accumulate the same test across several sends, or accept that only a large difference will be readable and treat everything smaller as a tie.
Treating it as a tie is a real result, by the way. It means the subject line was not the constraint, and the next thing worth testing is who you are sending to. If placement is the actual problem, no subject line rescues it, and the diagnosis is different.
Hold some back
The point of a test slice is that there is something left to send the winner to.
Split a portion of the list, send A and B to that portion, wait, then send the winner to the remainder. A test that consumes the entire list optimises nothing, because the campaign is over by the time you know which version was better. You have bought a data point for the next campaign at the cost of sending half your list the worse email.
In SEMAOS the winner goes to the remainder automatically once the test window closes, or on your approval if you would rather look first. The product tour shows the broadcast screen.
Wait longer than feels necessary
Opens arrive over hours, not minutes, and they do not arrive uniformly.
A large share of eventual opens land in the first few hours, and the rest trickle in over the following day or two, but the exact curve depends on your list and your send time enough that it is worth measuring your own rather than trusting anyone's average. Reading a test at thirty minutes measures which recipients happened to be at their desk when it arrived. That correlates with time zone and job function, not with your subject line.
For a B2B list, four hours is the minimum and overnight is better. Mail that arrives at 9am gets read after lunch by a meaningful share of the list, and those people are not randomly distributed between your variants unless you waited for them.
FAQ
How big does each variant need to be?
Roughly 1,500 per variant, if you want to trust a 5-point difference in open rate. That is a standard power calculation at a 40% baseline, not a rule of thumb. Smaller lists can still test, but only larger differences will be readable, and anything inside the margin should be treated as a tie.
How long should I wait before picking a winner?
Hours, not minutes, and longer for a B2B list where mail sent in the morning is read after lunch. Reading at thirty minutes measures who was at their desk, not which subject line worked.
Can I test the subject and the body at the same time?
You can send it, but you cannot learn from it. If B wins you will not know which change caused it, and you will carry the wrong lesson into the next campaign.
Should I optimise for opens or clicks?
Subject tests are read on opens because that is what a subject line controls. Body tests should be read on clicks or replies. Judging a body test on open rate measures nothing at all, since both variants had the same subject.