How do you reduce the risk of following false information when it comes to your split testing plan?
The best option is to actually perform an A/A test, which means you compare one variation against itself to make sure the tool, is tracking effectively. A/A testing is also the most efficient way to confirm the data you are following is accurate and can also mitigate false-positive tests.
The problem with A/A testing is that it takes extra time—time that most companies don’t have to spare.
So what do you do if A/A testing is not an option and you want to ensure you are on the right path?
- Backup tracking
It should be part of your standard operating procedure that you have backup tracking for your split-testing plan. One effective option is to pass your testing data to a tool such as Google Analytics. This will allow you to have a different perspective on the data to determine accuracy and efficiency.
- Limit your variations
The more variations you have in your test the more room there is for error.
The more you try to test at one time can lead to errors in data as well as take much longer to conclude the results from the test.
This goes the same for multivariate testing as well as A/B testing.
We always recommend that you limit what you are testing on any given campaign. This allows for a couple of factors.
- Less moving parts to cause errors
- The faster you can get to statistical significance
- Make sure you have a big enough sample size
Your test is showing as it is a big winner and it can be exciting leading you to want to push that “winner” to your site but what you need to pay attention to is the sample size.
When we are performing tests for our clients at Conversion Fanatics we never even start looking at the data from a test until we have at least 1000 unique visitors per variation, have at least 25 conversions per variation, or it hasn’t been running for a full calendar week.
Tuesday traffic will convert differently than Saturday traffic so you need to make sure you have at least one weekend in the test results.
If at all possible let your test run to significance but when that isn’t possible (some tests take forever to reach significance). When this isn’t an option make sure you have a big enough sample size and enough conversions to justify the results.
We can’t count how many times a test showed promise to be a big winner only to end up losing by a significant margin as the test grew more mature.
So to recap on what you can do to measure the accuracy of your test is to 1. Run an A/A test 2. Leverage backup tracking 3. Limit variations 4. Have a big enough sample size.