If you’re interested in creating an experiment plan, then you already know the value of conversion rate optimization, which can improve UX and increase revenue.
And the way those objectives are achieved is often the result of A/B tests determining the winning variations of design elements and copy choices.
Figuring out what to test and when can be a challenge, though, as the process should be strategic and not a guessing game. Therefore, if you want to meet your ecommerce conversion goals, you need to have a plan for A/B testing and designing your experiments.
Now, especially if you’re new to this, you may be wondering how do I make an experiment plan?
For starters, you need to think long-term and examine data about your website and visitor behavior and prioritize test ideas according to your goals and timelines.
When you put in the work up front to create a testing or experiment plan, you end up saving time and resources in the long run. Let’s take a closer look at how you can make an experiment plan for your business.
Why An Experiment Plan Is Essential For Your Business
Experiment plans help you test more effectively. They convey the need for certain tests along with a means of using a strategic, evidence-based approach to making optimization decisions about your website.
Experiment plans also make test-related communication among your team easier and more straightforward, as everyone involved can reference the plan, stay up-to-date, and manage their expectations.
How To Make A Basic Experiment Plan
A basic experiment plan requires four main parts:
- hypothesis
- setup
- variations
- results.
But before you can settle on a hypothesis, you need to prioritize a list of ideas you want to test. The minimum detectable effect (MDE) helps you do this. (You can learn more about MDE here.)
Creating Your Hypothesis
Once you know what you want to test, you need to create a hypothesis, which is typically structured as a “change and effect” statement and includes a reason why a specific change is likely to produce a particular effect.
UX expert Craig Sullivan also suggests structuring your hypothesis as follows:
- “Because we saw (data/feedback)
- We expect that (change) will cause (impact)
- We’ll measure this using (data metric).”
Designing Your Experiment’s Setup
When it comes to your experiment’s setup, you need to determine…
- where and when to run the experiment and for how long (e.g. on your homepage immediately after the page loads for two weeks)
- who will see the experiment (which target audience)
- the primary metric(s) you’ll use to measure success (e.g. revenue, new subscribers, etc.)
- and how much traffic you want to enter each variation.
Outlining Each Variation
If you’re A/B testing, you’ll have two variations. But if you’re multivariate testing, you’ll have more.
For each variation, you need to assess how it tests your hypothesis, determine what elements it will contain, and decide if it will be a redirect experiment (aka a split URL experiment).
Assessing Your Experiment’s Results
At the end of your experiment, you need to examine your results and determine the key insights, projected ROI, and whether your primary metric(s) for success achieved statistically significant results.
Assessing statistical significance is key and has a lot to do with your sample size. Keep in mind that while a test may yield a winning result, if your sample size was too small, that result may not be statistically significant. In other words, your data may be insufficient and caused by chance. Consequently, you’d be taking a bigger risk if you were to act on it.
At the end of your experiment plan, you should include details about the next steps you’ll take after running the experiment. These steps should fall in line with your business’ goals and timelines.
What do you think of this post? Does your business use A/B testing plans? Contact us via social media and share your thoughts.