Are A/A Tests Useful Or Ineffective Time Guzzlers?

Last Updated on

Before you run, you walk. With any process, there is an order. And in that vein, before you A/B test, you should A/A test. In other words, you should test the original variation of your website against itself to see if conversion rates are more or less the same as a way to establish a reliable baseline. You should also conduct A/A tests to double-check the accuracy of whichever A/B testing software you’re using.

Well, this is what a lot of marketers will tell you anyhow. But are A/A tests actually worth the time and effort?
Before we delve into that, however, let’s first cover the various types of A/A tests you can perform.

Knowing Your A/A Testing Options

There are three main types of A/A tests:

A/A – 50/50 Split

This is the most common type of testing setup, with the goal being to validate the setup of the test by making sure each variant yields roughly the same results. “You’re testing the same thing against itself, to see if there’s noise in the data, instead of signal,” says Craig Sullivan.

aa-testing

Image Source

A/A/B/B – 25% Splits

With this type, the tests are split into four samples, containing “both A and B in duplicated segments.” According to Sullivan, this form of testing is once again designed to validate the test setup but its second aim is to “confirm if there are oddities in the outcomes.”

The biggest downsides of this type of test are 1. the effect of skew is more prominent, 2. the sample size is smaller, and 3. each sample’s error rate is higher.

A/B/A

This type of test also helps identify setup issues but without taking up as much testing time. Like, A/A/B/B, A/B/A uses smaller A samples, which, as a result, produces higher error rates. This type of testing also necessitates a more complex reporting interface.

Why You Should Run A/A Tests

aa-testing

Image Source

Running A/A tests is generally regarded as “statistics good practice,” and just what you do if you want to make sure you’re testing “properly.” Other compelling reasons include…

  • A/A tests tell you when a test is significantly biased in some way.
  • They are a good way to have a clean dry-run before you start with the game-changers.
  • They help you spot inaccuracies in A/B testing software – check for “a statistically significant (>95% statistical significance) difference between the control and variation,” suggests Optimizely.

What Are The Downsides Of A/A Testing?

craig-sullivanCraig Sullivan“For me, the problem is always eating real traffic and test time, by having to pre-load the test run time with a period of A/A testing.  If I’m trying to run 40 tests a month, this will cripple my ability to get stuff live.  I’d rather have a half-day of QA testing on the experiment than run 2-4 weeks of A/A testing to check it lines up.

The other problem is that nearly 80% of A/A tests will reach significance at some point.  In other words, the test system will conclude that the original is better than the original with a high degree of confidence! Why?  Well, it’s a numbers and sampling thing but also because you’re reading the test wrong.  If you have small samples, it’s quite possible that you’ll conclude that something is broken when it’s not.”

So What’s The Bottom Line?

A/A testing definitely has its merits but it’s also a time-guzzler, and time is money after all. Plus, depending on the sample size, the results may not be super precise. A better alternative is to open up your segmentation.

Additionally, you can set up a multi-step QA process, which is much quicker to do. And, according to Brooks Bell, it “includes multiple people (to negate a single point of failure) acting on the conversion point in a live test environment, and confirmation of the conversion being tracked in a live data environment.”

What do you think about A/A testing? Is it worth it? Or, are the alternatives more effective?