In our industry, we talk about email A/B testing so frequently that we assume all email marketers are executing them. The truth is only 31% of brands test a majority of their emails and 39% don’t test their emails at all. Why? Because it’s easier not to.
While we can understand the hesitation, email testing is still the best way to continually refine your email campaign strategy over time. Testing takes personal preference out of email campaign optimization, and instead uses data to drive your decision-making. Marketers that fail to run email A/B tests are not running at their optimal potential, and brands that do A/B test see 21% more revenue.
Keep reading as we address the top challenges that marketers face with email A/B testing, and we’ll explain how you can tackle them head on to gain a competitive edge.
Challenge 1: Knowing What Email A/B Test To Run
Either you’ve exhausted your ideas of what to test, or you’ve got so many ideas that you’re struggling with how to prioritize. The best approach is to start with the end goal—what element you’re attempting to optimize—then establish the email testing methodology that will prove (or disprove) your theory.
Prioritize – Before beginning a test, measure the potential impact and how much time and effort will go into getting the test implemented. The ideal test is one that yields the most email campaign optimizations and requires minimal effort to build. Start small with email subject line testing, and build from there. If you need additional email testing inspiration, see our blog on Email Testing: Beyond the Subject Line.
Build on past tests – If you’re out of new email testing ideas, don’t worry–not all A/B tests need to be new. Email campaign optimization is an ongoing process, and your next test can be built on your learnings from past tests. As an example: if you’ve already tested trigger timing for the first touch of a cart abandon series, consider testing the time between follow-ups.
Break down the hard tests – Not all tests are created equally, and some tests are more complex to execute than others. To make sure you don’t continue to sideline your harder email tests, break them down into smaller hypotheses that you can test individually. All of the smaller outcomes still provide critical insight into the larger goal.
Challenge 2: Setting Up a Valid Email Test
The purpose of the test is to gather meaningful data, so take your time before the test begins to establish the most effective email testing methodology.
Isolate a single variable – A/B test one variant at a time, otherwise you can’t be sure what was responsible for any changes in performance. It also pays to be patient–repeat your test a few times to normalize data and ensure consistency. Do everything you can to reduce external noise so you’re not making changes to your email program that were influenced by incorrect drivers.
Sample size – If your audience is too small, then you’re relying somewhat on luck or large variance, ultimately leading to a failed test. Run the numbers that each variation will potentially yield, and if you’re not able to draw meaningful data, then move on to another email test.
Determine key KPIs – Make sure you’ve identified a testing metric that measures your desired outcomes. Those tracking parameters need to be included in the test from the beginning. You can’t analyze what you don’t measure.
Email testing tools – Utilize the tools available to you for quicker and easier A/B testing. Most email service providers have built-in testing tools where you can quickly build two email variations, and have a winner automatically determined. You’ll need to measure statistical significance manually, but Neil Patel’s A/B Testing Significance Calculator makes it easy.
Challenge 3: Interpreting the Data
You’ve set up your email tests, and now it’s time to kick up your feet and reap the benefits, right? Not quite–now you’ll have to report on campaign results and pull insights from them. Follow these steps to turn your data into insights:
1. Review your theory – Before you analyze the data, revisit your hypothesis and the metrics that would prove your theory. Review the prediction you made prior to launching the email test so you can remain focused on the main goal.
2. Identify expected (or unexpected) behavior – Determine whether or not your email A/B test produced the intended results. Or, did something make you say, “hmm…”? Any shift in behavior, intended or unintended, is valuable information that helps you uncover insights. Look for patterns and trends across multiple tests of the same variable to avoid acting on some fluke that possibly occurred during one email test.
3. Add context – After you’ve connected steps 1 and 2, add specific details from your email testing methodology to draw meaning from your test. For example, let’s say you have a chart of email sign-up rates during your email capture popup A/B test, and one variation is clearly trending higher than the other. The chart itself implies no actionable takeaway, and it’s the context and explanation of your results that make more of an impact. In this example, describe the A/B test variations and which variance you’ve deduced to be more effective based on the higher sign-up rates.
4. Incorporate your learnings – If you have high confidence in a clear takeaway, then proceed with applying the winning strategy to your email program. To capitalize on your learnings, consider where else in your email program these results could be relevant whether that is another email campaign or another iteration of a similar email test.
Challenge 4: Not Seeing Results
Often times, we run email tests but our engagement numbers don’t reflect any big improvements. Fewer than 20% of marketers that A/B test produce statistically significant results 80% of the time. We then ask ourselves, “What are we doing wrong?” If you’re regularly getting inconclusive results, then rework your email testing strategy with these tips:
Make bolder changes – With a large audience, small differences can still produce meaningful results. Brands with smaller audiences, on the other hand, need to focus on bigger differences between variations to drive bigger wins.
Persistence and iterative testing – Not seeing a lift in engagement is still a lesson in and of itself. You can make changes and add variety to your email program without negative implications. You can also build on your initial test if there was supporting data, but not enough statistical significance. Fine-tune your testing methodology, build on your learnings, and test again with an improved version.
Remain actionable – Analyzing data won’t do you any good if you are unable to translate your testing efforts into successful actions. First, you’ll need a clear understanding of your email test findings so that you can present the information in an easily digestible and compelling way. Empower your stakeholders with the information to develop an action plan, and act quickly to push the winning experience to your audience.
Conclusion
The concept of email A/B testing may be simple, but to execute and learn from a test can be daunting. Even so, email testing remains the tried-and-true approach for optimizing email marketing strategies, and should not be neglected. Power through the challenges to stay competitive, and drive business growth and innovation.
