4 A/B Testing Mistakes You Are Probably Making

Last Updated on

You say you are testing and working on improving your conversion rates, but most likely you are making some mistakes? After thousands of tests we have boiled it down to these 4 common mistakes people make when looking at conversion optimization.
1. False Positives
You have a 6 way A/B test running, looking to improve some engagement on your site by changing the headlines. Quickly you see a clear winner, but you stay the course gathering more data to make sure you have a clear winner. The test reaches 95% conclusive thinking you really have stumbled upon something great, you decide to run with the changes only to be left scratching your head when the increase wasn’t an increase at all.
What happened?
That my friends is what we like to call a false positive. Everything looks great but other factors played a role in the outcome of the test such as people, other steps of the funnel etc…
To remedy this, cut the losers and retest the winning variable against the control again in order to confirm you truly have a winner. This will take a little extra time but it will be worth it.
2. Not knowing when to say when
We see it all too often. You setup a test but it seems to take forever to reach conclusive results. None of the variables seem to be showing much promise but you still sit on the test waiting for it to magically turn into an improvement. Plenty of data is being collected but still not a true winner.
What we recommend here is to cut ties with the test and turn your focus on elements that could yield more of an impact. It is ok to fold your hand on tests if they aren’t showing improvement to focus on other elements that will.
For example we had been sitting on a test for two weeks with one client and the stats just didn’t seem to be changing much. Plenty of conversions on each of the variables but still no clear winner. So we decided to stop that test and turn focus to other places where we can see some distinct improvement.
3. Testing Too Many Small Elements
Oh believe me, there is a time and place for incremental changes that can yield big results. We have loads of those case studies. But sometimes what you need is an entirely new look and feel.
For sake of example, we had a client campaign that was getting dismal results on their lead generation. We tested a completely new landing page and the results were over a 300% increase on one traffic source.
Once you make the big change, then start testing backwards, working on the incremental changes to dial in your conversion rates even further.
4. Not Sticking With It
This is a big one. You run a few tests, see some improvement, then you turn your attention to something else. In order to have full control of your conversion rate optimization you need to continually be testing. Markets change, people change, and so should your marketing efforts.
Even the slightest improvement is great and we are firm believers in compounding conversions, the slight edge on continuous improvement. Sometimes you will have big winners, sometimes small ones but it is important that you keep at it as it will pay big dividends in the long run.
Bonus: Wrong focus
Thought we would throw out a little bonus element here. In the sake of your testing you need to make sure you are focusing on the RIGHT metrics as not all improvements will make you more money.
Instead of focusing on the single page conversion rate, keep your focus on your actual earnings per click and customer value. After all those are the elements that drive out profit.
All other elements should lead to the bigger goal and that is an increase in customer value. It is not always a fast path but it is a rewarding one.
It is our hope that you aren’t making these common mistakes and thanks for reading.