How Many Conversions Is Enough Before Declaring A Split-Test Winner?

The amount of times we get asked this question is actually pretty shocking, a valid question and one that needs to be addressed. Today we are going to discuss how many “visitors” or “conversions” you need before declaring a split test a winner.

When running split tests, no matter what software or method you are using, we typically focus on the main objective first and that is more times than not conversions.
A conversion can mean many different things, it all depends on your specific needs, KPIs, and goals. Examples of conversions could be, a form was submitted, a specific page was visited, product added to the cart, purchase made, upsell purchased, just to name a few.  Again this depends on your needs and what you are looking to improve.

Secondary to watching for conversions we look at the overall visitor stats per variation and the actual statistical significance of the particular test and its variations.

But how many is enough?

Each campaign is different and as mentioned each objective and goals are different as well.
If you are looking for a general rule of thumb here is what we look for…

1.  Statistical significance

The basic definition is…  A result that is not likely to occur randomly, but rather is likely to be attributable to a specific cause.  

Statistical significance no matter how good your calculations or software is comes with a certain degree of error.  We almost never base our results solely on significance due to this margin of error.

The general rule of thumb is to run tests to 90%+ significance as a baseline.  This does help alleviate some of the margins for error.  You may find that if measuring for significance that it takes an extended period of time of letting the test run before reaching significance.  Of course the higher the percentage of significance the better but we usually keep our baseline at 90%.

Depending on how you are measuring you may also find that variations of a test will reach significance faster than others, which usually occurs when you have a variation that shows a relatively high improvement percentage.

2.  Number of visitors

There has been some debate over this particular topic.

How many visitors should you run to each variation before you can determine a winner?  Again as with significance, there can be a big margin for error and for that we don’t recommend relying solely on visitors as your choice of measurement of a variation winner.

Some people say that you should run 200 visitors per variation and others say you should run 2000.  Who is right?

Well, both are technically right.  Again you need to pay attention to what your objectives are and what your particular scenario is.

If you are working on a low traffic volume and low conversion volume 200 visitors may be a big enough sample size.  Whereas if you have a larger volume 2000+ may be the right number.

As a general rule of thumb, we don’t start looking at test results until we see at least 500 visitors per variation and don’t really start thinking about making a change until we see 1000+ visitors per variation.  This helps again reduce the margin for error and the chance at getting a false positive which could lead you in the wrong direction with your optimization plans.

3.  Number of conversions

As mentioned above “conversions” can mean several different things but the two most common are lead or sale conversion.

When looking for winning variations the most common element we use is the number of conversions.  It only makes sense.  If you are optimizing for the generation of more leads you would want to see which variation actually gives you more leads right?

The number of visitors and statistical significance, although important, doesn’t matter as much as how many “conversions” you actually have.

This is the biggest determining factor of whether you have a winner or not.

The general rule of thumb here, and this can change slightly depending on volume, is looking for a minimum of 25 conversions per variation before we start taking a hard look at what to do next.

Anything less than that and you can have big swings in which variation is the winner.  For example, if one variation has 10 conversions and is converting at 10% and another variation has 11 conversions, just one more conversion on either of the variations can change the result dramatically.

When getting over 25 conversions per variation that margin for error shrinks dramatically.  Of course the more the better but you can generally start seeing a winning variation when getting over 25 conversions.

So how many is enough?

The short answer is that it depends on your volume.  We have given some general rules we go by that may help you find your next winning variation.  Use these three measurements collectively to achieve the maximum effectiveness of your tests.

Be sure to like, comment, share and subscribe.

If you want more out of your digital marketing efforts be sure to reach out as we are happy to help point you in the right direction.