Amazon

Amazon Manage Your Experiments (Beta) A/B Testing for A+ Content

By Tinuiti Team

Nov 05 2019

Amazon recently announced its new “Manage Your Experiments” (MYE) feature. This new tool allows sellers to A/B test their A+ content to make data-driven decisions that can increase sales.

According to Amazon, “Experiments can help you statistically find the best A+ content for your listings, improve future content, and help drive more sales.”

There are three steps to A/B testing your A+ content:

Experiment: Running controlled experiments to test the performance of your A+ content
Analyze: Interpreting results backed by statistics to make data-driven decisions about your A+ content
Optimize: Maximizing sales by launching high-performing A+ content that resonates with your customers

Once you interpret your results and optimize your content, it’s time to A/B test something new. Done right, here’s what the A+ content experimentation lifecycle should look like:

amazon-manage-your-experiments

A/B testing best practices

Here are some do’s and don’ts when it comes to A/B testing your A+ content through Amazon’s new Manage Your Experiments feature.

Do:

Experiment on high traffic ASINs. The more people who visit your product details page, the larger the sample size for your experiment.
Make one big change at a time. If you change too many variables at once, you won’t know which ones influence any changes in your results.
Plan a long enough test to make a high confidence decision. You can run experiments for 4, 6, 8, or 10 weeks.

Don’t:

Test minor differences in content. Focus on changing headlines, images, and other meaningful elements of your A+ content.
End your experiment early. Gather as much data as you can to make the most informed decisions possible.
Apply different ASINs to your two versions of content. To run your test, both versions of content are required to have the exact same ASINs applied.

What elements can you test?

You can A/B test variations of nearly any element of your A+ content. Here are some A+ content test ideas from Amazon Brand Services:

Add a comparison table
Add or remove a company/brand logo
Move your existing A+ modules around
Update or replace the text in your modules
Swap the order of your images
Present the same images and text using a different module
Try lifestyle images
Introduce new A+ content modules

“The Manage Your Experiments Beta allows Vendors and Sellers to work towards optimizing their A+ content design. You can be very intentional with elements of the design you would like to test and apply learnings to content you create in order to develop a foundation for your optimal product detail page experience. This feature is an absolute game changer and everyone in the industry is extremely excited to start experimenting.”

– AJ Swamy, Manager, Creative Strategy at Tinuiti

aj-swamy-tinuiti

How to run an A/B test through Manage Your Experiments

There are four main parts of running your A/B test for A+ content through MYE:

Create a B variation of your A+ content
Set up your experiment
Run your experiment
Measure your results

Part 1: Create a B variation of your A+ content

Create both versions of your content in the A+ Content Manager. Apply the same ASINs to both Version A and Version B. You must have the exact same set of ASINs applied to both versions of content in order to run the experiment.

Next, submit your content to Amazon for approval. This can take up to 7 business days (but usually happens within 48 hours).

Part 2: Set up your experiment

Once Amazon approves your content, it’s quick and easy to set up your experiment. First, select which content you’d like to test based on the ASIN. Then, set up and schedule your test through the MYE feature.

Part 3: Run your experiment

Once your experiment goes live, each customer who visits your detail page is randomly assigned a content version. Logged in customers will see the same variation every time they visit the product detail page.

While your experiment is in progress, you can change the duration or cancel the experiment (though we recommend against the latter). You can’t edit your A+ content while experiments are in progress or scheduled to begin.

Part 4: Measure your results

A few weeks after your experiment begins, you can start seeing data on each variation and how well they’re performing. Preliminary data can be misleading, though, so let your experiment run its course before you interpret the results.

Once your experiment ends, Amazon gives you:

Recommendations on which content variation is more effective
How confident you should be in the recommendation
The confidence interval of likely outcomes from that content
The estimated 12-month impact on sales

These performance insights can help you choose the most effective content for your product detail pages which, in turn, can boost sales and conversions for your Amazon products.

Ready to get started with Manage Your Experiments for A+ content? Sellers can access the feature through Seller Central, while vendors can head over to Vendor Central to begin experimenting.