Subject Line A/B Tests: How Not to Fool Yourself
Why most subject line A/B tests prove nothing
Most subject line A/B tests measure noise, not a difference between variants. Two versions go out to groups that are too small. Someone checks the stats after two hours and declares a winner. A few days later the same subject loses to the variant it beat before. That isn’t knowledge about your audience. It’s a reading of random fluctuation.
The same mistakes keep coming back: a sample that’s too small, stopping the measurement before a full cycle, comparing two sends from different days. On top of that, the metric itself is leaky. These days, opens are distorted by mailbox providers fetching images before a person even opens the message.
The brain attaches a cause to every difference - that’s how we’re wired. That’s why a result has to be checked by method, not by gut feeling. Looking into send queues shows that send order matters. Some “winning” variants won because of the order they went out in, not because of the subject line. One variant can go out earlier, hit a better window for recipients and win without any credit to the copywriter.
What you’re really measuring when you test a subject line
The subject line affects the open. An open is not a business goal. There’s a whole chain between the send and the money, and any link in it can break the result.
- Delivery - the recipient’s server accepted the message.
- Inbox placement - it landed in the main inbox, not in spam or the promotions tab.
- Open - the noisiest metric.
- Click - counted on unique recipients, more resistant to automated content fetching.
- Conversion - a purchase, a sign-up, a booked call.
- Reply - the strongest engagement signal, for filters too.
An intermediate metric can lie in a very specific way. An aggressive subject line lifts opens and, along the way, lifts spam complaints. You see the first in the report right away. The second only shows up in later campaigns. Spam complaints and unsubscribes are a deferred cost, paid for with sender reputation. A winning test followed by three sends that land worse was never a win.
Conditions without which a subject line test is worthless
The list split has to be random. Split by ID, by sign-up date or alphabetically, and you’re testing a segment, not a subject line. Your oldest subscribers behave differently from people who signed up last week, so the difference you see will describe the history of your list. That’s all.
The remaining conditions come down to one thing: both variants have to take the exact same technical path.
- Random assignment of recipients to groups, with no sorting by any attribute.
- One changed element - the subject line. Body and preheader stay untouched.
- Same sender domain, same From address, same Return-Path, same IP pool.
- A shared time window with interleaved sending, not variant A in the morning and variant B in the afternoon.
- Minimum group size set before the start, not picked after the fact.
- A fixed time for reading the result, the same for both groups.
Tip: write down the success criterion and the reading time before you hit send. A criterion set after seeing the data will always bend toward the result you like.
Infrastructure that skews test results more than copywriting
SPF, DKIM and DMARC have to be correct for both variants. If one goes out from a different host or with a different signing setup, you’re comparing authentication, not subject lines. The same goes for PTR and Return-Path alignment with the sender domain. Filters look at the whole configuration before they even get to the text in the subject line.
Warmup and real group size
A fresh IP address in warmup has different delivery behavior than a pool with history. A test during warmup is unreliable, because the infrastructure variable moves faster than the content effect. Calculate results on delivered messages, not sent ones - that’s the basics. Hard and soft bounces change the real size of the groups, sometimes unevenly.
Blocks during sending
A block at a major filter can stop one variant halfway through the queue. The other keeps going and “wins” for reasons unrelated to the content. When you’re recovering from a block, it’s worth pausing subject line tests. Delivery changes from hour to hour, so any measurement from that period describes the state of the filter, not how recipients reacted.
How to read the result and when to accept there’s no difference
“No difference” is a valid result. It means the tested element doesn’t decide anything, and it lets you stop fiddling with it. Treating it as a failure leads to stretching the data until something starts to look like an advantage.
Start by learning your own noise level. An A/A test: you send an identical subject line to two random halves of the same list. The difference you see between them is your natural fluctuation. Any A/B result smaller than that fluctuation means nothing. Even if it looks nice in the report.
And don’t add up many small tests into one conclusion if they differed in list, timing or message content. That isn’t a bigger sample, just a patchwork of different experiments. One winning subject line doesn’t make a rule either. A rule appears when the same mechanism repeats on a different segment and in a different month.
What to test so your conclusions carry over
Test mechanisms, not individual sentences. A specific subject line’s win won’t help in your next campaign, because you won’t use that subject line again. What helps is knowing how subject length works, whether the company name is included, personalization, a question versus a statement, or a number in the subject line.
Treat the preheader as a separate variable. Sometimes it matters more than the subject line itself, because it takes up more space in the preview. Check emoji and all caps more broadly than through opens - look at unsubscribes and spam complaints in the same send. Engagement-based segmentation also shifts the result: inactive subscribers react completely differently from people who click regularly. You can handle list splits, send variants and reports in the campaign features module, without stitching it together from several tools.
- Write the hypothesis in one sentence.
- Set the decision metric and the minimum group size.
- Randomize the split and prepare both variants with a single difference.
- Send in one time window, interleaving the variants.
- Read the result at the agreed time, on delivered messages.
- Record the decision, including “inconclusive”.
- Repeat on another segment before you treat it as a rule.
Tip: keep a test log with the hypothesis, conditions and result. Record inconclusive tests as carefully as the wins.
Consent, law and list hygiene as the backdrop for every test
Without consent there’s no clean test. GDPR and Article 398 of the Polish Electronic Communications Law require consent for commercial communication, in B2B as well. A business email address doesn’t exempt you.
A purchased list ruins every measurement. You mix people who never chose anything with people who signed up on purpose. The two groups react so differently that the effect of the subject line gets lost in that gap. On top of that, spam complaints go up and weigh on sender reputation long after the experiment ends. Keep the sign-up source on the side of the system where you actually collect contacts. That’s why sign-ups from forms and your store are connected through ready-made integrations, instead of uploading files of unknown origin.
Unsubscribing has to work in one click, in every test variant. The List-Unsubscribe header reduces spam complaints, because it gives recipients an easier way out than the “report spam” button. Legal compliance isn’t a separate department next to deliverability - it’s part of it. We build the MailCraft email marketing platform around exactly that idea.
Summary: fewer tests, run better
A meaningful test takes work before the send, not after it. After the send, all that’s left is reading the result and being honest about the numbers you see.
Three things to remember: a random list split, one variable at a time, a success criterion set in advance. The rest is discipline and patience.
We’re upfront about our own limitations. Some things we’d like to offer in the dashboard are on the roadmap, and that’s how we describe them, instead of selling them as ready. Being honest about your own data gives you more than yet another subject line variant.
Build your own test log. Other people’s “best practices” were made on a different list, different infrastructure and in a different period. Your conclusions, repeated on your own audience, are the only material worth basing decisions on.
