Email A/B testing means sending two versions of the same email to comparable slices of your list, changing one thing, and letting the numbers show which version your subscribers respond to. Done properly, it replaces opinion with evidence: you learn which subject lines earn the open, which calls to action earn the click and which offers earn the order. This guide covers what to test and in what order, how big your audience needs to be, which metric should pick the winner, and how to avoid the mistakes that make results meaningless.
- Change one variable at a time, write down a hypothesis first, and choose the winning metric before you press send.
- Start with what shows in the inbox (sender name, subject line, preheader), then move to copy, layout and calls to action.
- Aim for at least 1,000 recipients per version, and for ecommerce judge winners on placed orders where you can, not only opens.
- Record every result and reuse it across flows, paid social and your website, then retest as audience habits shift.
- For UK lists, testing never replaces PECR compliance: consent and opt-out rules apply to every variant.
Table of Contents
- What email A/B testing is and why it works
- What to test first: a priority order
- How to set up a test that you can trust
- Choosing the metric that decides the winner
- Common mistakes that make results unreliable
- From result to rollout: using what you learn
- Our perspective: test the system, not just the subject line
- Where to run tests in an ecommerce email programme
- Getting more from testing with the right support
- FAQ
- Sources
What email A/B testing is and why it works
Email A/B testing, often called split testing, compares two versions of an email to see which performs better with your audience. Salesforce describes the approach as a way to gather data on what resonates, so you can refine messaging, design and timing to lift open, click-through and conversion rates over time. The outcomes it measures are binary: a subscriber opens or does not, clicks or does not.
That simplicity is the strength. You are not asking customers what they prefer, which is unreliable. You are watching what they do, at scale, in the real inbox, with real money at stake.
The gains can be material. Econsultancy’s PureGym case study reports that a personalised campaign informed by message testing delivered a 69% increase in open rates, an 89% increase in click-through rates and a 206% higher conversion rate than emails sent to all users. That is one company’s result, not a promise for yours, but it shows why testing sits at the centre of any serious email programme.
For ecommerce brands there is a second reason. Email is one of the few channels where you own the audience, so every improvement compounds across every send, every automated flow and every product launch. A subject line that lifts opens by a few points on a weekly campaign keeps paying for the rest of the year.
What to test first: a priority order
You can test a long list of elements: sender name, subject line, preheader, email copy, image size, calls to action, number of links, send time and email length. TechTarget’s guidance recommends beginning with sender name, subject line and preheader, because recipients see these before they open, and improving opens decides whether the rest of the email is seen at all. Then move on to copy, headers and calls to action.
Match the variable to the weak point in your funnel:
| Where the problem is | What to test | Metric to watch |
|---|---|---|
| Low opens | Sender name, subject line, preheader, send time | Open rate |
| Healthy opens, weak clicks | Body copy, headers, layout, call to action, number of links | Click-through rate |
| Clicks but few orders | Offer, product selection, landing page match, urgency | Placed order rate, revenue per recipient |
TechTarget makes the same point: if opens are healthy but click-through is weak, test body copy rather than the subject line.
Inbox elements: sender, subject line and preheader
The sender name is the cheapest test and often the most revealing. Research from the US Consumer Financial Protection Bureau, reported on Digital.gov, found that emails from a personal-looking account often perform better than those from a generic address. For a retailer, that might mean testing your brand name against a named person at the brand.
For subject lines, test one idea at a time: a benefit against a curiosity hook, a short line against a longer descriptive one, or the presence of a product name. Do not change tone, length and personalisation together, or you will not know what moved the result.
The preheader is the most neglected line in email. A UK example: a MarketingExperiments test with Electronics Weekly sent preheader variants to 5,000 subscribers over three weeks, keeping sender, subject line, timing and campaign type constant. The winning version produced a 4.9% higher open rate, a 30% relative increase, and a click-through rate of 3.77% against 3.20%. Note the discipline as much as the result: everything else was held still, so the change could be credited to the preheader.
Content: copy, length, layout and calls to action
Once opens are healthy, look inside the email. Klaviyo’s A/B testing guide lists sender name, subject line, layout, copy, call to action and send time as the elements to test systematically. Useful content tests for retailers include:
- A single hero product against a grid of three or four.
- A button call to action against a text link.
- Short, punchy copy against longer, descriptive copy.
- Lifestyle imagery against plain product shots.
- A discount led by percentage against one led by a fixed amount.
Treat other people’s winners with caution. Nutshell’s guide points out that whether short or long copy works depends on the industry and the complexity of the product, so a result from another brand does not automatically transfer to yours.
Personalisation and segments
Some of the strongest tests are not about a single element but about who receives what. Comparing a generic message with one tailored to purchase history, browsing behaviour or lifecycle stage is where testing meets segmentation. Tools for automated customer segmentation make it easier to build comparable groups, which matters because a test is only fair when each version goes to similar people.
Post-purchase and review emails
Transactional and post-purchase emails get high attention and are excellent test candidates. If you ask for product reviews, the timing, subject line and copy of the request are all testable, and this breakdown of email review request timing and copy gives you a sensible set of variants to start from.
How to set up a test that you can trust
- Write a hypothesis. Econsultancy’s testing and optimisation report stresses a hypothesis that explains why you expect a result. Kath Pay’s formula is to state that you believe making change X will cause Y to happen because of Z. The “because” is what turns a result into a lesson.
- Pick one variable. Everything else stays identical: audience, send time, offer and sender, unless one of those is the thing under test.
- Choose the winning metric in advance. Deciding after the fact invites you to pick whichever number flatters your favourite version.
- Define the audience and split. Randomise the split from a comparable segment so you are not comparing your best customers with lapsed ones.
- Set the duration. Decide how long the test runs before the result is read.
- Send, wait and read the result. Do not peek after an hour and stop early.
- Document it. Record the hypothesis, versions, audience size, result and what you will do next.
How big should the audience be?
Small tests produce noise. HubSpot recommends sending an A/B-tested email to at least 1,000 contacts, and Klaviyo advises at least 1,000 recipients for each version to avoid inconclusive results. TechTarget adds that larger samples are generally more accurate and suggests choosing a sample big enough to reach 95% confidence.
The Econsultancy report describes statistical significance as a 90% to 95% chance that the result reflects the conditions of the experiment, rather than randomness or bias in who was selected. In practice, if a result is within a hair’s breadth and your sample is small, call it a draw and move on to a bolder test.
If your list is smaller than 2,000 subscribers, you have two honest options. Test bigger changes, because large differences show up in small samples, or test across several sends of the same type and pool the results.
How long should it run?
In HubSpot, you set a test length in hours, after which the remaining recipients get the winning version. That suits opens and clicks, which arrive quickly. Purchases are slower. Klaviyo’s note on testing for purchase rate says it recommends a longer window when sales are the metric, so customers have time to buy before a winner is declared. For a considered purchase such as furniture or electronics, leave more time than you would for a fast-moving fashion drop.
Choosing the metric that decides the winner
Common winning metrics are open rate, click rate and click-through rate, as listed in HubSpot’s documentation. They are useful, but they are not the business outcome. Match each to what the test was designed to change:
- Open rate for sender, subject line and preheader tests. Treat it with care, because privacy features that prefetch email content can inflate opens, so read it alongside clicks.
- Click-through rate for copy, layout and call-to-action tests.
- Placed order rate or revenue per recipient for offer, product and flow tests. Klaviyo lets you choose placed order rate as the winning metric, which links the email message directly to sales.
A subject line that wins on opens but attracts the wrong clicks can lose on revenue. For an ecommerce brand, the safest habit is to name a primary metric and a guardrail: for example, win on click-through rate, but only if unsubscribe rate does not rise.
Common mistakes that make results unreliable
- Testing too many variants at once. The CFPB experience reported on Digital.gov is that too many versions spread segments thin and can stop a test reaching significance. Two or three variants is plenty for most lists.
- Changing several things in one variant. If the subject line, hero image and offer all differ, a win teaches you nothing.
- Stopping early. Early leaders often fade. Fix the duration beforehand.
- Testing trivial changes. A different shade of button rarely moves revenue. Test ideas that reflect a real difference in message or offer.
- One and done. The same CFPB piece warns against treating a result as permanent, because audience preferences evolve. Revisit important tests periodically.
- Ignoring context. A winning Black Friday subject line may fail in February. Note the season and the offer next to every result.
- Forgetting compliance. In the UK, email marketing is governed by PECR, and the ICO’s direct marketing by email guidance, updated on 28 April 2026, says organisations must apply the relevant consent or soft opt-in rules and respect objections and opt-outs. Every variant must meet them, including the unsubscribe route.
From result to rollout: using what you learn
After the test, the winner is determined by the metric you selected beforehand and sent to the rest of the audience. The value comes next. Econsultancy’s report says findings should be recorded and used to inform future campaigns, website copy and other digital channels, and Salesforce likewise advises applying insights to your wider email strategy.
A simple testing log does the job. Keep one row per test:
| Date | Hypothesis | Variable | Audience size per version | Metric | Result | Next action |
|---|---|---|---|---|---|---|
| Month, year | We believe a named sender will lift opens because it feels personal | Sender name | 4,000 | Open rate | Record the lift | Roll out to weekly campaigns, retest in six months |
Then carry lessons across channels. A subject line angle that wins in email is a strong candidate for a paid social headline or a product page title. Econsultancy notes that email findings can inform testing plans for web copy, search keywords, PPC and landing pages. The same logic works in reverse: winning ad concepts from a structured creative testing framework are good raw material for email tests.
Our perspective: test the system, not just the subject line
Most brands we speak to test email in isolation: a subject line here, a button there. The results are real but small, because email is one stage of a customer journey that also runs through paid media, creative, the website and post-purchase. At Evolve Commerce we treat testing as part of a connected growth system, where each channel feeds learning to the others.
Three principles guide how we think about it:
- Measure to revenue. An open rate is a signal, not a result. We want the test tied to orders and revenue, which is why attribution and analytics sit alongside the email programme rather than after it.
- Share winners across channels. An angle that earns clicks in a flow should be tested in ad creative, and the reverse. Our clients have achieved annual revenue growth between 130% and 450%, and a repeatable testing habit across channels is a large part of how that kind of growth is built.
- Prioritise by impact. Lifecycle flows such as abandoned cart and winback usually send to people with clear intent, so a modest improvement there is worth more than polishing a monthly newsletter.
The discipline is boring and it works: one hypothesis, one variable, a pre-set metric, a record of what was learned.
— Evolve Commerce
Where to run tests in an ecommerce email programme
Not every send deserves a test. Rank your programme by revenue and volume and begin at the top:
- Welcome series. Every new subscriber sees it, so small gains compound. Test the incentive, the sender and the first-email subject line.
- Abandoned cart and browse flows. High intent and high return. Test send timing, reminder copy and whether to feature the product image or social proof.
- Post-purchase flows. Test cross-sell timing, review requests and delivery messaging. See our notes on the post-purchase experience.
- Winback. Test the offer and the number of touches against a lapsed segment.
- Campaigns. Weekly and promotional sends give you quick tests on subject lines, hero content and send time.
SMS can be tested the same way, and the results often cross over. If you run both channels, see our plan for ecommerce SMS and cart recovery.
Getting more from testing with the right support
Testing well takes list hygiene, clean attribution and the time to analyse and act on results. If you would rather have a team run it as part of a wider growth programme, our Email & SMS Marketing service covers lifecycle automation and optimisation on platforms such as Klaviyo, and it sits alongside paid media, performance creative and SEO so that learning moves between channels. Pricing for managed services is agreed per engagement, so get in touch through our site to talk through your list and goals.
If you want to see the numbers behind your tests, our AdWize analytics platform brings channel data into one dashboard with server-side attribution. Plans are listed at AdWize pricing: Starter at £49 per month, Accelerator at £149 per month and Scale at £249 per month.
FAQ
Can you A/B test on a website as well as in email?
Yes. The method is the same: one change, a comparable audience, a pre-chosen metric. Email results can also inform web tests, since Econsultancy notes that findings can feed testing plans for web copy, search keywords, PPC and landing pages. Many teams test the subject line promise in email and then check that the landing page repeats it.
How many subscribers do I need for an email A/B test?
Both HubSpot and Klaviyo point to around 1,000 recipients per version as a sensible floor. With fewer, test larger changes or pool results across several similar sends. Always read the result against your confidence threshold, typically 95%, before declaring a winner.
What should I test first in an email?
Start with what is visible before the email is opened: sender name, subject line and preheader. Improving opens decides whether the rest of the email gets seen. Once opens are healthy, move on to body copy, layout and the call to action.
Should I pick the winner on open rate or on sales?
Pick it on the metric that matches the change you are testing. Opens suit subject lines, clicks suit content, and placed orders suit offers and flows. For an online shop, revenue is the number that matters, so allow enough time for purchases to come in before you decide.
How often should I rerun a test?
Revisit important results periodically rather than treating them as permanent. Audience preferences change, and a finding from a different season or a different customer mix may no longer hold. A good rule is to retest your highest-volume flows at least twice a year.
Sources
- Email Marketing A/B Testing: A Complete Guide, Salesforce
- PureGym case study, Econsultancy
- 4 email A/B testing best practices, TechTarget
- A/B testing your emails, Digital.gov
- Preheader testing case study, MarketingExperiments
- Guide to A/B testing email marketing campaigns, Klaviyo
- What Is Email A/B Testing, Nutshell
- Email Marketing Best Practice: Testing and Optimisation, Econsultancy
- Run A/B tests for marketing emails, HubSpot
- A/B test for purchase rate, Klaviyo
- Guidance on direct marketing using electronic mail, ICO
