A/B Testing

Overview

This guide includes everything needed to use the A/B Testing feature in the "Beamable SDK for Unity".

The purpose of this feature is to allow game makers to deploy new functionality to subset of players.

What is A/B Testing?

A/B Test trials are experiments that can be created in the Portal to group players into buckets either by percentage (e.g. "50% of all users") or according to criteria (e.g. Stat of PLAYER_LEVEL > 1).

Benefits of A/B Testing

Game makers often know exactly which decisions make the best experience for players. When in doubt, game makers can use an A/B Testing trial to try multiple solutions before finalizing the game design.

Increase KPIs - Key performance indicators, including retention, can be isolated and improved through testing
Increase Revenue - By improving KPIs, improve revenue
Decrease Costs - Game makers can focus on a leaner feature-set which is proven to matter to players
Decrease Risks - Roll out new gameplay features more frequently to a subset of the players. Ensure its ready before releasing to everyone

Flowchart

Here is a flowchart demonstrating an example of one A/B Testing Trial.

The game front-end loads the player manifest from the TrialDataService. The PlayerProgression data returned depends on a player-specific Stat value.

Glossary

Here are some common terms:

Name	Detail
Acceptance Strategy	Determines how players qualify for a trial. Values; • GLOBALLY_OPEN - Indicates that once a player qualifies for this trial, they will be part of it until the trial is (even if ejected they cease to meet the qualifying criteria in the future) • GLOBALLY_OPEN_WITH_FLOW - Behaves like a GLOBALLY_OPEN trial, except that a player will be removed from the Trial if they no longer qualify for it • MUTUAL_EXCLUSIVE - Indicates that even if a player qualifies for this trial, they will only join it if they are not already part of another trial that is `MUTUAL_EXCLUSIVE` or `MUTUAL_EXCLUSIVE_WITH_FLOW`. Once a player joins this trial, they are a part of it until it is (even if they ejected cease to meet the qualifying criteria in the future) • MUTUAL_EXCLUSIVE_WITH_FLOW - Behaves like a MUTUAL_EXCLUSIVE trial, except that a player will be removed from the Trial if they no longer qualify for it
Allocation	Determines how players qualify for a specific cohort within a trial. Values; • % Distribution - Specify a percentage of the total player population for each cohort • Custom - Specify any player stat and a relational operator. This determines the cohort (e.g. stat of `PLAYER_LEVEL` > 1)
Cohorts	A grouping of members
Cohort Data Overrides	Specifies the Game Base Cloud Data references that is overridden (i.e. changed) for a given cohort Note: Any text/binary format is compatible with Game Base Cloud Data. Many game makers choose YAML. See Wikipedia's YAML for more info
Lifecycle	Determines the current lifecycle status of the trial. Values; • Running - Members receive data overrides. Players can join/leave the trial • Paused - Members receive data overrides. Players cannot join/leave the trial • Ejected - Members receive no data overrides. Players cannot join/leave the trial
Member	Refers to a player within a trial
Member Count	How many members are in a given cohort

API

Unlike many Beamable Features, A/B Testing does not require a specific Beamable Feature Prefab to be used. The main entry point to this feature is C# programming.

Here are API highlights for beamableAPI.TrialDataService.

Method Name	Detail
GetGameManifest	Loads the generic game dataset excluding any player-specific overrides
GetPlayerManifest	Loads the specific game dataset including any player-specific overrides

🚧
Gotchas
Here are hints to help explain some of the trickier concepts:
• The downloaded manifest is compressed in Gzip format. Be sure to account for that
• The url of the cloud data leaves the protocol unspecified. Be sure to prepend either "http://" or "https://"

{
   "result":"ok",
   "meta":[
      {
         "sid":1335576733310978,
         "version":1,
         "ref":"game_world_config",
         "uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
         "cohort":{
            "trial":"",
            "cohort":""
         }
      }
   ]
}

{
   "result":"ok",
   "meta":[
      {
         "sid":1335576733310978,
         "version":1,
         "ref":"game_world_config",
         "uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
         "cohort":{
            "trial":"",
            "cohort":""
         }
      }
   ]
}

Stats

The Stats feature is related to A/B Testing in a couple of ways.

1. Automatic Stats

Each player who is a member of an A/B Testing Trial automatically has these populated Stats.

These values can be accessed programmatically via set as game.private.player or set in the Beamable Portal.

Name	Detail
`TRIALS`	This one stat contains a list of all trials. The stat value is dynamically built Ex. value `[New Trial.Group 2]`
Ex. `trialmember:New Trial`	There is an additional stat per trial. The stat name is dynamically built as well as the stat value Ex. value `Group 2`

2. Criteria Stats

For any A/B Testing Trial with an "Allocation" of type "Custom", the game maker specifies any player Stat and a relational operator. This determines the cohort (e.g. stat of PLAYER_LEVEL > 1).

These values can be accessed programmatically via get as game.private.player.

Adding A/B Testing

Game Maker User Experience

During development, the game maker's user experience is as follows:

Steps

📘
Related Guides
A common use case for the feature is covered in the guides.
• See Adding A/B Testing for more info

Advanced

This section contains any advanced configuration options and workflows.

Debugging Manifests

For debugging the manifests, game players can use the Admin Flow to enter commands.

Commands

Name	Detail
`CLOUD-MANIFEST`	Retrieve and output to the console log the game manifest, which includes the entire cloud data domain. Note: This is invokable only by a privileged user (i.e. C# Microservice OR admin user). So if privileged user invoking in Unity, be sure your runtime user is an admin of the realm – you can login to such a user using the Account Management Flow prefab
`CLOUD-PLAYER`	Retrieve and output to the console log the player manifest, which includes the entire cloud data domain.

📘
Best Practices
These hints make efficent use of concepts and workflows.
Unity Client

Trial assignment occurs on session start

Fetch the cloud data manifest again if the user changes

C# Microservice

Fetch the manifest with every incoming request, it is cheap to do so

Cache the cloud data in local storage or in memory