Let’s say that you’re a developer/publisher focused on improving your D1 retention (for example). How many different variables are affecting that stat?
Take your time, draw some diagrams, make an educated guess, accept that you’re wrong, and then get into the bin. The real answer to that question is terrifyingly large and impossible to attain.
With so many factors impacting the player experience and game performance, any attempts at optimization need to be tested rigorously in very specific conditions. They need to be A/B tested.
What is an A/B Test?
An A/B test essentially compares different versions of the same variable to determine the impact of changes to that variable. The idea is to run concurrent tests in which conditions are as close to identical as possible, save for the variable being tested.
The control group continues to experience the ‘default’ behavior and the treatment group receives the new behavior, so that you can compare the results and determine the impact of changing your chosen variable.
Why A/B Test?
Testing changes in this way is important for several reasons:
- By testing changes on a small portion of your population, you minimize the impact of rolling out negative change.
- You can statistically prove the effects of your proposed change, instead of relying on gut feeling.
- By deciding on what constitutes ‘significant’ change before running the test, you can be objective in your assessment.
TL:DR – If you want to have any understanding of how to iteratively improve your game, you need to A/B Test the heck out of it.
How to perform a successful A/B Test
It’s an iterative process. The diagram below breaks down A/B Testing step-by-step.
Patience and discipline are key to running a successful A/B test. Little by little, those sweet (sweet) results will roll in. Rome wasn’t built in a day, friends.
How to perform an unsuccessful A/B Test
There are plenty of ways to completely invalidate your results, but these are the ones that make the podium:
🥉- Fiddle as you go
Letting your motivations and emotions affect significance will ruin everything. It doesn’t matter if you have a silver-bullet-eureka-moment or regret setting your standards so high – once the test begins, you HAVE to let it run its course.
🥈- Test multiple changes at once
As soon as multiple variables are at work, you can’t accurately evaluate the precise impact of any of them. It’s one at a time, or not at all.
🥇- Use different audiences
You might think that a handy way to divide your groups is by geolocation, demographic, or any other recorded characteristic. Wrong, wrong, a thousand times WRONG. Your control and treatment groups must be made up of identical audiences or the whole exercise is pointless.
A/B Tests in deltaDNA
We already have A/B testing functionality within the platform and its vital to optimizing everything from game difficulty to offer value, but that’s the tip of the testburg. We’re in the process of building something altogether inspiring – the Lord of the Rings to our current Hobbit. Watch this space for more updates!