Recently, we talked about important metrics to track when you’re A/B testing.
But before you can get into tracking, there’s the matter of creating your hypothesis and designing the test.
Figuring out what to test isn’t just a bunch of guesswork. Figuring out what to test and how to test it requires a structured, research-based approach.
So if you’re interested in A/B testing and are seeking an answer to the question how do I make a hypothesis and design a single test, keep reading below.
How To Create A Hypothesis
A hypothesis is a prediction you want to test, usually formed as a “change and effect” statement. For example, If I change the button color from red to blue, email signups will increase.
A hypothesis also typically includes a reason why you think a particular change will produce a particular outcome.
Here’s the most basic structure you can follow when creating a hypothesis:
If _____(variable to be changed), then _____ (effect produced) because _____(reason why).
UX expert Craig Sullivan also suggests the following:
- “Because we saw (data/feedback)
- We expect that (change) will cause (impact)
- We’ll measure this using (data metric).”
Choosing A Variable
If you’re A/B testing, as opposed to multivariate testing, you should isolate a single variable. Choose a variable you can control that your site’s visitors interact with and is therefore likely to have an effect on visitor behavior.
To narrow your variable search (and make a more informed decision), try the following:
- Analyze visitor or customer feedback, looking for information about points of friction on your website.
- Use heatmaps and other visitor recording tools to asses behavior patterns.
- Send surveys to current customers asking why they chose to do business with you and if there were anything that could enhance the experience.
Predicting A Result
Look at the data you have about your variable of choice to establish a baseline metric. Use your baseline to make a prediction about your test’s results. And when the test is over, compare your results against your baseline to measure the effect.
Inform Your Rationale
Once again, A/B testing does not rely on guesswork and neither does any part of creating a hypothesis to test. Accordingly, your rationale should be informed with research—for example, quantitative and qualitative research about your visitors’ behavior (heatmaps, eye-tracking, surveys) or other tests involving a similar change that yielded similar results.
How To Design A Single Test
Once you have your hypothesis, you’re ready to set up your test. You’ll need to set up two versions of your landing page: one that’s unaltered and another that includes the change to the variable you want to test.
Next, you’ll need to split your traffic randomly into equal groups (most testing tools can do this for you). Then, you’ll have a few factors you need to determine:
- Your sample size
- The length of your test
- Statistical significance (how confident you need to feel about your results before you choose one variation over another).
How you determine these factors can vary, depending on your testing tool and the type of your A/B test. But, generally speaking, the length of your test impacts the size of your sample. And the more statistically significant your results are, the more confident you can feel about your results’ accuracy.
Here are some tips about improving your A/B test’s design:
- Test both variations at the same time.
- Run your test long enough to yield helpful data.
- Collect feedback from visitors via surveys or polls during the test.
- Don’t lose sight of your goal metric, or the outcome you predict the test will produce.
And when the test is complete, use an A/B testing calculator to determine the significance of your results.
A/B testing can be complicated. But if you take it one step at a time and don’t lose sight of your testing goals, you can achieve accurate, winning results.
What do you think of these tips for creating a hypothesis and designing a single test? Get in touch via social media and share your thoughts!