Code

Allow game makers to deploy new functionality to subset of players

Overview

Here is everything needed to use the A/B Testing feature in the "Beamable SDK for Unity".

The purpose of this feature is to allow game makers to deploy new functionality to subset of players.

What is A/B Testing?

A/B Test trials are experiments that can be created in the Portal to group players into buckets either by percentage (e.g. "50% of all users") or according to criteria (e.g. StatStat - Active data entity used for read/write of player state of PLAYER_LEVEL > 1).

Benefits of A/B Testing

Game makers often know exactly which decisions make the best experience for players. When in doubt, game makers can use an A/B Testing trial to try multiple solutions before finalizing the game design.

  • Increase KPIs - Key performance indicators, including retention, can be isolated and improved through testing
  • Increase Revenue - By improving KPIs, improve revenue
  • Decrease Costs - Game makers can focus on a leaner feature-set which is proven to matter to players
  • Decrease Risks - Roll out new gameplay features more frequently to a subset of the players. Ensure its ready before releasing to everyone

Flowchart

Here is a flowchart demonstrating an example of one A/B Testing TrialTrial - An experiment to group players into buckets either by percentage or according to a player stat criteria.

The game front-end loads the player manifest from the TrialDataService. The PlayerProgression data returned depends on a player-specific StatStat - Active data entity used for read/write of player state value.

Glossary

Here are some common terms.

Name

Detail

Acceptance Strategy

Determines how players qualify for a trial. Values;

GLOBALLY_OPEN - Indicates that once a player qualifies for this trial, they will be part of it until the trial is (even if ejected they cease to meet the qualifying criteria in the future)
GLOBALLY_OPEN_WITH_FLOW - Behaves like a GLOBALLY_OPEN trial, except that a player will be removed from the Trial if they no longer qualify for it
MUTUAL_EXCLUSIVE - Indicates that even if a player qualifies for this trial, they will only join it if they are not already part of another trial that is MUTUAL_EXCLUSIVE or MUTUAL_EXCLUSIVE_WITH_FLOW. Once a player joins this trial, they are a part of it until it is (even if they ejected cease to meet the qualifying criteria in the future)
MUTUAL_EXCLUSIVE_WITH_FLOW - Behaves like a MUTUAL_EXCLUSIVE trial, except that a player will be removed from the Trial if they no longer qualify for it

Allocation

Determines how players qualify for a specific cohort within a trial. Values;

% Distribution - Specify a percentage of the total player population for each cohort
Custom - Specify any player stat and a relational operator. This determines the cohort (e.g. stat of PLAYER_LEVEL > 1)

Cohorts

A grouping of members

Cohort Data Overrides

Specifies the Game Base Cloud Data references that is overridden (i.e. changed) for a given cohort

Note: Any text/binary format is compatible with Game Base Cloud Data. Many game makers choose YAML. See Wikipedia's YAML for more info

Lifecycle

Determines the current lifecycle status of the trial. Values;

Running - Members receive data overrides. Players can join/leave the trial
Paused - Members receive data overrides. Players cannot join/leave the trial
Ejected - Members receive no data overrides. Players cannot join/leave the trial

Member

Refers to a player within a trial

Member Count

How many members are in a given cohort

API

Unlike many Beamable FeatureFeature - An individual aspect of the Beamable product used to create a great user experiences, A/B Testing does not require a specific Beamable Feature PrefabFeature Prefab - The drag-and-drop template for a specific Beamable product feature to be used. The main entry point to this feature is C# programming.

Here are API highlights for beamableAPI.TrialDataService.

Method Name

Detail

GetGameManifest

Loads the generic game dataset excluding any player-specific overrides

GetPlayerManifest

Loads the specific game dataset including any player-specific overrides

🚧

Gotchas

Here are hints to help explain some of the trickier concepts.

• The downloaded manifest is compressed in Gzip format. Be sure to account for that
• The url of the cloud data leaves the protocol unspecified. Be sure to prepend either "http://" or "https://"

{
   "result":"ok",
   "meta":[
      {
         "sid":1335576733310978,
         "version":1,
         "ref":"game_world_config",
         "uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
         "cohort":{
            "trial":"",
            "cohort":""
         }
      }
   ]
}
{
   "result":"ok",
   "meta":[
      {
         "sid":1335576733310978,
         "version":1,
         "ref":"game_world_config",
         "uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
         "cohort":{
            "trial":"",
            "cohort":""
         }
      }
   ]
}

Stats

The Stats feature is related to A/B Testing in a couple of ways.

1. Automatic Stats

Each player who is a member of an A/B Testing TrialTrial - An experiment to group players into buckets either by percentage or according to a player stat criteria automatically has these populated StatStat - Active data entity used for read/write of player states.

These values can be accessed programmatically via set as game.private.player or set in the Beamable Portal.

Name

Detail

TRIALS

This one stat contains a list of all trials. The stat value is dynamically built

Ex. value [New Trial.Group 2]

Ex. trialmember:New Trial

There is an additional stat per trial. The stat name is dynamically built as well as the stat value

Ex. value Group 2

2. Criteria Stats

For any A/B Testing TrialTrial - An experiment to group players into buckets either by percentage or according to a player stat criteria with an "Allocation" of type "Custom", the game maker specifies any player StatStat - Active data entity used for read/write of player state and a relational operator. This determines the cohort (e.g. stat of PLAYER_LEVEL > 1).

These values can be accessed programmatically via get as game.private.player.

Adding A/B Testing

Game Maker User Experience

During development, the game maker's user experience is as follows.

Steps

📘

Related Guides

A common use case for the feature is covered in the guides.

• See Adding A/B Testing for more info

Advanced

This section contains any advanced configuration options and workflows.

Debugging Manifests

For debugging the manifests, game players can use the Admin Flow to enter commands.

Commands

Name

Detail

CLOUD-MANIFEST

Retrieve and output to the console log the game manifest, which includes the entire cloud data domain.

Note: This is invokable only by a privileged user (i.e. C# Microservice OR admin user). So if privileged user invoking in Unity, be sure your runtime user is an admin of the realm – you can login to such a user using the Account Management Flow prefab

CLOUD-PLAYER

Retrieve and output to the console log the player manifest, which includes the entire cloud data domain.

📘

Best Practices

These hints make efficent use of concepts and workflows.

Unity Client

  • Trial assignment occurs on session start
  • Fetch the cloud data manifest again if the user changes

C# Microservice

  • Fetch the manifest with every incoming request, it is cheap to do so
  • Cache the cloud data in local storage or in memory

Did this page help you?