Cold Email A/B Testing: What to Test First (and What Not To)

Most cold email A/B testing fails not because the wrong variable was tested, but because the sample size never reached statistical relevance before someone declared a winner. The variables below are ordered by how much they typically move reply rates, paired with realistic sample-size expectations.

The Priority Order for What to Test

1. Offer / angle — the single biggest lever. Testing two genuinely different value propositions or pain-point framings against each other typically produces the largest measurable difference of anything on this list.
2. Subject line — meaningfully moves open rate, and open rate is a leading indicator for reply rate, though the relationship isn't perfectly linear. See our subject lines guide for the categories worth testing against each other.
3. CTA type (soft vs hard ask) — often produces a bigger swing than most people expect, especially by sequence position; see our soft ask vs hard ask guide.
4. Email length — worth testing once the above are stable, since length interacts with the offer and audience in ways that are hard to predict without testing.
5. Sender name / persona — worth testing for larger teams with multiple reps, since some names and personas measurably outperform others for reasons that aren't always predictable in advance.

Sample Size: The Constraint Most People Ignore

A cold email campaign with a 5% baseline reply rate needs several hundred sends per variant before a meaningful difference becomes statistically distinguishable from noise — testing two subject lines against 40 recipients each and declaring a winner because one got 3 replies and the other got 1 is not a valid test, even though it feels like one. Most cold-email A/B tests that get treated as conclusive were actually underpowered, which means teams frequently "optimize" based on noise rather than signal.

What's Usually Not Worth Testing

• Minor wording tweaks within an otherwise identical email — the effect size is typically too small to distinguish from noise at realistic cold-email volumes.
• Send time down to the specific hour — day-of-week and rough time-of-day (morning vs afternoon) can matter, but testing 9am vs 9:30am specifically is rarely worth the sample size it would take to resolve.
• Font, color, or HTML formatting choices — cold email generally performs better as plain-text-style formatting regardless, making this an even lower-priority test than it would be for marketing email.

How to Actually Read the Results

Beyond raw sample size, watch for confounding variables that make a test invalid even with enough volume: sending variant A on Monday and variant B on Thursday introduces a day-of-week confound; testing two subject lines but changing the body copy at the same time makes it impossible to attribute the result to either variable specifically. Isolate one variable at a time, run both variants concurrently against comparable audience segments, and let the test run long enough to reach a sample size where the difference is unlikely to be explained by chance alone.

FAQ