A/B Testing - Code

Allow game makers to deploy new functionality to subset of players [BUSI-ABTesting-03]

Overview

This guide includes everything needed to use the A/B Testing feature in the "Beamable SDK for Unity".

The purpose of this feature is to allow game makers to deploy new functionality to subset of players.

What is A/B Testing?

A/B Test trials are experiments that can be created in the Portal to group players into buckets either by percentage (e.g. "50% of all users") or according to criteria (e.g. Stat of PLAYER_LEVEL > 1).

Benefits of A/B Testing

Game makers often know exactly which decisions make the best experience for players. When in doubt, game makers can use an A/B Testing trial to try multiple solutions before finalizing the game design.

  • Increase KPIs - Key performance indicators, including retention, can be isolated and improved through testing
  • Increase Revenue - By improving KPIs, improve revenue
  • Decrease Costs - Game makers can focus on a leaner feature-set which is proven to matter to players
  • Decrease Risks - Roll out new gameplay features more frequently to a subset of the players. Ensure its ready before releasing to everyone

Flowchart

Here is a flowchart demonstrating an example of one A/B Testing Trial.

The game front-end loads the player manifest from the TrialDataService. The PlayerProgression data returned depends on a player-specific Stat value.

Glossary

Here are some common terms:

NameDetail
Acceptance StrategyDetermines how players qualify for a trial. Values;

GLOBALLY_OPEN - Indicates that once a player qualifies for this trial, they will be part of it until the trial is (even if ejected they cease to meet the qualifying criteria in the future)
GLOBALLY_OPEN_WITH_FLOW - Behaves like a GLOBALLY_OPEN trial, except that a player will be removed from the Trial if they no longer qualify for it
MUTUAL_EXCLUSIVE - Indicates that even if a player qualifies for this trial, they will only join it if they are not already part of another trial that is MUTUAL_EXCLUSIVE or MUTUAL_EXCLUSIVE_WITH_FLOW. Once a player joins this trial, they are a part of it until it is (even if they ejected cease to meet the qualifying criteria in the future)
MUTUAL_EXCLUSIVE_WITH_FLOW - Behaves like a MUTUAL_EXCLUSIVE trial, except that a player will be removed from the Trial if they no longer qualify for it

AllocationDetermines how players qualify for a specific cohort within a trial. Values;

% Distribution - Specify a percentage of the total player population for each cohort
Custom - Specify any player stat and a relational operator. This determines the cohort (e.g. stat of PLAYER_LEVEL > 1)

CohortsA grouping of members
Cohort Data OverridesSpecifies the Game Base Cloud Data references that is overridden (i.e. changed) for a given cohort

Note: Any text/binary format is compatible with Game Base Cloud Data. Many game makers choose YAML. See Wikipedia's YAML for more info
LifecycleDetermines the current lifecycle status of the trial. Values;

Running - Members receive data overrides. Players can join/leave the trial
Paused - Members receive data overrides. Players cannot join/leave the trial
Ejected - Members receive no data overrides. Players cannot join/leave the trial

MemberRefers to a player within a trial
Member CountHow many members are in a given cohort

API

Unlike many Beamable Features, A/B Testing does not require a specific Beamable Feature Prefab to be used. The main entry point to this feature is C# programming.

Here are API highlights for beamableAPI.TrialDataService.

Method NameDetail
GetGameManifestLoads the generic game dataset excluding any player-specific overrides
GetPlayerManifestLoads the specific game dataset including any player-specific overrides

🚧

Gotchas

Here are hints to help explain some of the trickier concepts:

• The downloaded manifest is compressed in Gzip format. Be sure to account for that
• The url of the cloud data leaves the protocol unspecified. Be sure to prepend either "http://" or "https://"

{
   "result":"ok",
   "meta":[
      {
         "sid":1335576733310978,
         "version":1,
         "ref":"game_world_config",
         "uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
         "cohort":{
            "trial":"",
            "cohort":""
         }
      }
   ]
}
{
   "result":"ok",
   "meta":[
      {
         "sid":1335576733310978,
         "version":1,
         "ref":"game_world_config",
         "uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
         "cohort":{
            "trial":"",
            "cohort":""
         }
      }
   ]
}

Stats

The Stats feature is related to A/B Testing in a couple of ways.

1. Automatic Stats

Each player who is a member of an A/B Testing Trial automatically has these populated Stats.

These values can be accessed programmatically via set as game.private.player or set in the Beamable Portal.

NameDetail
TRIALSThis one stat contains a list of all trials. The stat value is dynamically built

Ex. value [New Trial.Group 2]
Ex. trialmember:New TrialThere is an additional stat per trial. The stat name is dynamically built as well as the stat value

Ex. value Group 2

2. Criteria Stats

For any A/B Testing Trial with an "Allocation" of type "Custom", the game maker specifies any player Stat and a relational operator. This determines the cohort (e.g. stat of PLAYER_LEVEL > 1).

These values can be accessed programmatically via get as game.private.player.

Adding A/B Testing

Game Maker User Experience

During development, the game maker's user experience is as follows:

Steps

📘

Related Guides

A common use case for the feature is covered in the guides.

• See Adding A/B Testing for more info

Advanced

This section contains any advanced configuration options and workflows.

Debugging Manifests

For debugging the manifests, game players can use the Admin Flow to enter commands.

Commands

NameDetail
CLOUD-MANIFESTRetrieve and output to the console log the game manifest, which includes the entire cloud data domain.

Note: This is invokable only by a privileged user (i.e. C# Microservice OR admin user). So if privileged user invoking in Unity, be sure your runtime user is an admin of the realm – you can login to such a user using the Account Management Flow prefab
CLOUD-PLAYERRetrieve and output to the console log the player manifest, which includes the entire cloud data domain.

📘

Best Practices

These hints make efficent use of concepts and workflows.

Unity Client

  • Trial assignment occurs on session start
  • Fetch the cloud data manifest again if the user changes

C# Microservice

  • Fetch the manifest with every incoming request, it is cheap to do so
  • Cache the cloud data in local storage or in memory