A/B Testing - Guide

[BUSI-ABTesting-02]

Overview

This guide includes everything needed to use the A/B Testing feature in the "Beamable SDK for Unity".

The purpose of this feature is to allow game makers to deploy new functionality to subset of players.

Game Maker User Experience

During development, the game maker's user experience is as follows:

Steps

Follow these steps to get started:

Step 1. Create Data

Here the data will be created. This represents the default data which all users will receive (unless overridden by the Trial created per Step 2).

StepDetail
1. Open the Portal• See beta-portal.beamable.com
2. Open "Game Based Cloud Data"
3. Upload Data

• Click "Upload"
• Populate all data fields
• Optional, choose the appropriate file from Data Files below
• Click "Upload"

*Note: At present the only data format supported for "Cloud Data" governed by Trials is YAML

Step 2. Create Trial

Here the Trial will be created. This represents the rules for if/when the default data created in Step 1 above will be overridden.

🚧

Private Stats

Note that Trials only work with private game stats. These are categorized under the game.private.player namespace.

StepDetail
1. Open the Portal• See beta-portal.beamable.com
2. Open "Trials"
3. Create the Trial

• Choose "Allocation" of "Custom" and add 2 cohorts
• Populate all data fields
• Click "Create"

Note: See A/B Testing - Code (Glossary) for more info

4. Assign the Data• Assign the override data that will be loaded instead of the default data from Step 1 above
• Optional, choose the appropriate files from Data Files below
5. Play the Trial

• Click "Play"
• Click "Confirm"

Note: See A/B Testing - Code (Glossary) for more info

6. Set the Stat

• Open "Player Administration"
• Open "Stats"
Enter the PlayerId of the active player from the Unity Console Window and the namespace of game.private.player
• Click "Add Player Stat"
• Enter Name of PLAYER_LEVEL and a Value of 1 or 2. Each returns a dataset via TrialDataService



Note: The trial in this example depends on a stat value. However, other types of trials do not. Choose the best criteria for the needs of the project.


Step 3. Load Data

Here the game client will load the Trial data. If the current player qualifies for the criteria of the Trial created in Step 2, the player will receive overridden data. Otherwise the player will receive the default data created in Step 1.

3a. Load Manifest

GetCloudDataManifestResponse playerManifestResponse =
     await _trialDataService.GetPlayerManifest();

3b. Load Data

Loop through the meta. There will be 0 or more values. The implementation here depends on the needs of the project. In this example, each loop iteration is parsed as Json and tested for compatibility with the MyPlayerProgression data type.

foreach (CloudMetaData cloudMetaData in playerManifestResponse.meta)
{
    string response = 
        await _trialDataService.GetCloudDataContent(cloudMetaData);

    MyPlayerProgression myPlayerProgression = 
        JsonUtility.FromJson<MyPlayerProgression>(response);

    // Store the data
    if (myPlayerProgression != null)
    {
        _data.MyPlayerProgression = myPlayerProgression;
    }
}

Examples

📘

Beamable SDK Examples

The following example code is available for download at GitHub.com/Beamable_SDK_Examples

Flowchart

Here is a flowchart demonstrating an example of one A/B Testing Trial.

The game front-end loads the player manifest from the TrialDataService. The PlayerProgression data returned depends on a player-specific Stat value.

Data Files

To follow along with the steps above using the example data download these Json data files which matches the MyPlayerProgression data type. Or create and choose your own custom files and custom data type.

  • base_cohort.json - For use with Step 1.3 above
  • player_level_1_cohort.json - For use with Step 2.4 above
  • player_level_not1_cohort.json - For use with Step 2.4 above

Code

Here are examples which cover common programming needs.

The TrialDataServiceExample.cs loads the MyPlayerProgression object with appropriate values. The related A/B Test trial uses the player's Stat of PLAYER_LEVEL to determine the appropriate values.

using System.Collections.Generic;
using System.Threading.Tasks;
using Beamable.Common.Api;
using Beamable.Common.Api.CloudData;
using Beamable.Examples.Services.CloudSavingService;
using Beamable.Examples.Services.ConnectivityService;
using Beamable.Examples.Shared;
using UnityEngine;
using UnityEngine.Events;

namespace Beamable.Examples.Services.TrialDataService
{
    /// <summary>
    /// Represents the data in
    /// the A/B Testing trial.
    /// </summary>
    [System.Serializable]
    public class MyPlayerProgression
    {
        public int MaxHealth = 100;
        public int MaxInventorySpace = 10;
    }

    /// <summary>
    /// Holds data for use in the <see cref="ConnectivityServiceExampleUI"/>.
    /// </summary>
    [System.Serializable]
    public class TrialDataServiceExampleData
    {
        public bool IsUIInteractable = false;
        public string DataName = "MyPlayerProgression";
        public MyPlayerProgression MyPlayerProgression = null;
        public List<CloudMetaData> CloudMetaDatas = new List<CloudMetaData>();
        public bool IsInABTest { get { return CloudMetaDatas.Count > 0; } }
    }

    [System.Serializable]
    public class RefreshedUnityEvent : UnityEvent<TrialDataServiceExampleData>  { }

    /// <summary>
    /// Demonstrates <see cref="TrialDataService"/>.
    ///
    /// NOTE: This demo uses other concepts
    /// too. See <see cref="CloudSavingServiceExample"/>
    /// for more info.
    /// 
    /// </summary>
    public class TrialDataServiceExample : MonoBehaviour
    {
        //  Events  ---------------------------------------
        [HideInInspector] public RefreshedUnityEvent OnRefreshed = new RefreshedUnityEvent();

        //  Fields  ---------------------------------------
        private TrialDataServiceExampleData _data = new TrialDataServiceExampleData();
        private ICloudDataApi _trialDataService;
        private IBeamableAPI _beamableAPI;

        //  Unity Methods  --------------------------------
        protected void Start()
        {
            Debug.Log($"Start() Instructions...\n" +
                      " * Setup AB Testing in Portal per https://docs.beamable.com/docs/abtesting-code\n" +
                      " * Run The Scene\n" +
                      " * See onscreen UI for results.\n" +
                      " * If IsInABTest is false, something is incorrect. Repeat these steps.\n" + 
                      " * If IsInABTest is true, everything is correct. Visit the portal to change " +
                      "the `PLAYER_LEVEL` stat value, then repeat these steps see load other data.\n");

            SetupBeamable();
        }


        //  Methods  --------------------------------------
        private async void SetupBeamable()
        {
            _beamableAPI = await Beamable.API.Instance;

            Debug.Log($"beamableAPI.User.id = {_beamableAPI.User.id}");

            _trialDataService = _beamableAPI.TrialDataService;

            await LoadTrialData();
        }

        public async Task<EmptyResponse> LoadTrialData()
        {
            // Load any trials
            GetCloudDataManifestResponse playerManifestResponse =
                await _trialDataService.GetPlayerManifest();
            
            // Loop through trials
            _data.MyPlayerProgression = null;
            _data.CloudMetaDatas = playerManifestResponse.meta;
            foreach (CloudMetaData cloudMetaData in _data.CloudMetaDatas)
            {
                string path = $"http://{cloudMetaData.uri}";

                // Load the data, respecting GZip format
                string response = 
                    await ExampleProjectHelper.GetResponseFromHttpWebRequest(path);

                MyPlayerProgression myPlayerProgression = 
                    JsonUtility.FromJson<MyPlayerProgression>(response);

                // If trial is related, store data
                if (myPlayerProgression != null)
                {
                    _data.MyPlayerProgression = myPlayerProgression;
                }
            }

            _data.IsUIInteractable = true;
            Refresh();
            return new EmptyResponse();
        }


        public void Refresh()
        {
            string refreshLog = $"Refresh() ..." +
                                $"\n * IsInABTest = {_data.IsInABTest}" +
                                $"\n * CloudMetaDatas.Count = {_data.CloudMetaDatas.Count}" +
                                $"\n\n";

            //Debug.Log(refreshLog);

            // Send relevant data to the UI for rendering
            OnRefreshed?.Invoke(_data);
        }
    }
}

Verify Success

After running the scene containing the TrialDataServiceExample.cs per above, you can further verify success.

Visit the Portal and change the value of any stat related to the trial for the currently logged-in PlayerId. See the Unity console window for the PlayerId.

Steps

  • Change the value of the PLAYER_LEVEL stat to 1. Run the scene and see the result.
  • Change the value of the PLAYER_LEVEL stat to 2. Run the scene and see the result.

If the onscreen UI shows different text in each case, then all is setup properly.

Troubleshooting Tips

Here are some areas to double check if your testing is not working as intended.

  • Ensure the stats used for segmentation are game private (game.private.player namespace).
  • Some values from your trials may need to be tweaked to affect the intended player population. See the A/B Testing - Code (Glossary) for more info on these parameters.
  • Ensure the files uploaded to Game Base Cloud Data are able to be deserialized properly in the file format your game is expecting.