A/B Testing - Code
Allow game makers to deploy new functionality to subset of players [BUSI-ABTesting-03]
Overview
This guide includes everything needed to use the A/B Testing feature in the "Beamable SDK for Unity".
The purpose of this feature is to allow game makers to deploy new functionality to subset of players.
What is A/B Testing?
A/B Test trials are experiments that can be created in the Portal to group players into buckets either by percentage (e.g. "50% of all users") or according to criteria (e.g. StatStat - Active data entity used for read/write of player state of PLAYER_LEVEL
> 1).
Benefits of A/B Testing
Game makers often know exactly which decisions make the best experience for players. When in doubt, game makers can use an A/B Testing trial to try multiple solutions before finalizing the game design.
- Increase KPIs - Key performance indicators, including retention, can be isolated and improved through testing
- Increase Revenue - By improving KPIs, improve revenue
- Decrease Costs - Game makers can focus on a leaner feature-set which is proven to matter to players
- Decrease Risks - Roll out new gameplay features more frequently to a subset of the players. Ensure its ready before releasing to everyone
Flowchart
Here is a flowchart demonstrating an example of one A/B Testing TrialTrial - An experiment to group players into buckets either by percentage or according to a player stat criteria.
The game front-end loads the player manifest from the TrialDataService
. The PlayerProgression
data returned depends on a player-specific StatStat - Active data entity used for read/write of player state value.
Glossary
Here are some common terms:
Name | Detail |
---|---|
Acceptance Strategy | Determines how players qualify for a trial. Values; • GLOBALLY_OPEN - Indicates that once a player qualifies for this trial, they will be part of it until the trial is (even if ejected they cease to meet the qualifying criteria in the future) |
Allocation | Determines how players qualify for a specific cohort within a trial. Values; • % Distribution - Specify a percentage of the total player population for each cohort |
Cohorts | A grouping of members |
Cohort Data Overrides | Specifies the Game Base Cloud Data references that is overridden (i.e. changed) for a given cohort Note: Any text/binary format is compatible with Game Base Cloud Data. Many game makers choose YAML. See Wikipedia's YAML for more info |
Lifecycle | Determines the current lifecycle status of the trial. Values; • Running - Members receive data overrides. Players can join/leave the trial |
Member | Refers to a player within a trial |
Member Count | How many members are in a given cohort |
API
Unlike many Beamable FeatureFeature - An individual aspect of the Beamable product used to create a great user experiences, A/B Testing does not require a specific Beamable Feature PrefabFeature Prefab - The drag-and-drop template for a specific Beamable product feature to be used. The main entry point to this feature is C# programming.
Here are API highlights for beamableAPI.TrialDataService
.
Method Name | Detail |
---|---|
GetGameManifest | Loads the generic game dataset excluding any player-specific overrides |
GetPlayerManifest | Loads the specific game dataset including any player-specific overrides |
Gotchas
Here are hints to help explain some of the trickier concepts:
• The downloaded manifest is compressed in Gzip format. Be sure to account for that
• The url of the cloud data leaves the protocol unspecified. Be sure to prepend either "http://" or "https://"
{
"result":"ok",
"meta":[
{
"sid":1335576733310978,
"version":1,
"ref":"game_world_config",
"uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
"cohort":{
"trial":"",
"cohort":""
}
}
]
}
{
"result":"ok",
"meta":[
{
"sid":1335576733310978,
"version":1,
"ref":"game_world_config",
"uri":"dev-trials.beamable.com/1208935505932288/DE_1208935505932289/1335576733310978/1",
"cohort":{
"trial":"",
"cohort":""
}
}
]
}
Stats
The Stats feature is related to A/B Testing in a couple of ways.
1. Automatic Stats
Each player who is a member of an A/B Testing TrialTrial - An experiment to group players into buckets either by percentage or according to a player stat criteria automatically has these populated StatStat - Active data entity used for read/write of player states.
These values can be accessed programmatically via set as game.private.player
or set in the Beamable Portal.
Name | Detail |
---|---|
| This one stat contains a list of all trials. The stat value is dynamically built Ex. value |
Ex. | There is an additional stat per trial. The stat name is dynamically built as well as the stat value Ex. value |
2. Criteria Stats
For any A/B Testing TrialTrial - An experiment to group players into buckets either by percentage or according to a player stat criteria with an "Allocation" of type "Custom", the game maker specifies any player StatStat - Active data entity used for read/write of player state and a relational operator. This determines the cohort (e.g. stat of PLAYER_LEVEL
> 1).
These values can be accessed programmatically via get as game.private.player
.
Adding A/B Testing
Game Maker User Experience
During development, the game maker's user experience is as follows:
Steps
Related Guides
A common use case for the feature is covered in the guides.
• See Adding A/B Testing for more info
Advanced
This section contains any advanced configuration options and workflows.
Debugging Manifests
For debugging the manifests, game players can use the Admin Flow to enter commands.
Commands
Name | Detail |
---|---|
| Retrieve and output to the console log the game manifest, which includes the entire cloud data domain. Note: This is invokable only by a privileged user (i.e. C# Microservice OR admin user). So if privileged user invoking in Unity, be sure your runtime user is an admin of the realm – you can login to such a user using the Account Management Flow prefab |
| Retrieve and output to the console log the player manifest, which includes the entire cloud data domain. |
Best Practices
These hints make efficent use of concepts and workflows.
Unity Client
- Trial assignment occurs on session start
- Fetch the cloud data manifest again if the user changes
C# Microservice
- Fetch the manifest with every incoming request, it is cheap to do so
- Cache the cloud data in local storage or in memory
Updated 5 months ago