What is A/B Testing?
A/B testing is a controlled experiment that compares two versions of a game or app: a control (A) and a variant (B). You measure which one performs better on a metric you care about, like retention or conversion.
Unity has its own A/B testing service, built on Remote Config as part of Unity Gaming Services, so you can run a test without shipping a new build.
How does A/B testing work?
Players are randomly split into two groups: one sees the control (A), the other sees the variant (B), while everything else about their experience stays the same.
You then measure a chosen metric, such as conversion rate or retention, for each group and compare the results.
A reliable test needs a clear success metric decided before the test starts, and enough players in each group to trust the result rather than random noise.
How to run an A/B test
1. Pick a value to test
Choose something already controlled by a remote config, such as an XP threshold, a difficulty setting, or an IAP offer price.
2. Set a success metric
Decide in advance what "better" means for this test, such as session length or purchase rate. See Unity's guide on which KPIs to watch for an ad placement test.
3. Create the test
Define the control and variant values from the Unity Gaming Services dashboard, no new build required.
4. Let it run
Leave the test live long enough to collect a reliable sample from both groups.
5. Compare and decide
Review the results in the dashboard, then roll out the winning value or end the test.
How is A/B testing used?
Studios use A/B testing for two main things: tuning gameplay values like difficulty or rewards, and optimizing ad monetization like waterfalls and placements.
For gameplay, a common example is testing how much XP it should take to gain a level, comparing session length between the two groups to see which value keeps players engaged longer. See Unity's own A/B testing use case documentation for a full worked example.
For ad monetization, LevelPlay's A/B testing tool lets you test waterfall configurations, ad networks, and reward amounts directly from the mediation dashboard, no coding required.
What are the benefits of A/B testing?
- Replace guesswork with evidence: Settle design debates with real player data instead of opinion.
- Lower the risk of a bad change: Validate an idea on a subset of players before rolling it out to everyone.
- Ship without a new build: Test remote-config-driven values and roll out the winner immediately, no app update required.
What are the challenges of A/B testing?
- Needs enough players: A low-traffic game may need to run a test for a long time before the result is reliable.
- Only tests one change at a time cleanly: Testing several changes at once makes it hard to know which one caused the result.
- A win isn’t always permanent: Player behavior can shift over time, so a value that won once is worth re-testing periodically.
Frequently asked questions (FAQ)
How long should an A/B test run?
Long enough to reach a statistically reliable result, which depends on your traffic volume and the size of the effect you’re testing for. Low-traffic games generally need to run tests longer to collect enough data to trust the result.
Can I A/B test my game without shipping a new build?
Yes, if the value you’re testing is controlled through a remote config rather than hardcoded. Unity Gaming Services lets you create and adjust A/B tests on live values from a dashboard, without requiring players to update the app.