Skip to main content

A / B Testing

Minds uses Growthbook to conduct A/B testing experiments.

Creating an Experiment

Basic Info

To create an experiment, login to Growthbook, and go to the experiments tab. From there you can hit the "New Experiment" button.

Fill out the fields on the first page of the form:

  • Name: A concise and relevant name for your experiment.
  • Type: Leave this set to "Code" - we have our own wrappers around the SDK.
  • Tags: Select any relevant tags.
  • Hypothesis: Make your hypothesis? What do we think will happen with the experiment?
  • Data Source: Leave set to main warehouse.

Basic Info Panel

When you're done, hit next.

Variations

Here we will be naming and labeling the variations. The "Control" refers to the already exsiting system, and the name can remain as the default 'Control'. For the variation, give it a relevant name and description. Feel free to add more variations if you are testing more than one variant.

Variations Panel

Metrics and Targeting

Finally add the relevant fields for Metric and Targeting. At the time of writing it is not possible to add your own Metrics through the Growthbook UI, however in future this will be possible.

Metric and Targeting

When you're done, hit Save. You'll see that this will create your experiment in 'Draft' mode - this will allow others to review your experiment and make changes to it before you make it live. When you are ready, you can start the experiment

Starting an Experiment

When your experiment is ready, you can start it. In the window that opens before starting, you are giving inputs to provide a reason for starting the new phase, time to start, type of phase and importantly, traffic split, which will control what ratio of users see the Control vs your variant.

Starting an Experiment

You should then tie your experiment to a feature flag with the same key, adding an override rule to the feature flag

Override rule

Developing for an Experiment

Engine

To interact with experiments on the engine side, you will need to use the Experiments Manager.

$experimentsManager = Di::_()->get('Experiments\Manager');
$experimentsManager->setUser($loggedInUser);

// Using isOn for binary experiments:
if ($experimentsManager->isOn('my-cool-experiment')) {
// Cool backend experiment
}

// Using hasVariation, which can be used for ternary experiments:
if ($experimentsManager->hasVariation('my-cool-experiment', true)) {
// Cool backend experiment
}

Front

On the front-end you have a few options - firstly you can use the run or hasVariation functions of the ExperimentsService. This can be used as such:

// Using the run function:
public shouldRenderCoolFeature(): boolean {
const output = this.experiments.run('my-cool-experiment');
return output === true;
}

// Using the hasVariation function:
public shouldRenderCoolFeature(): boolean {
return this.experiments.hasVariation('my-cool-experiment', true);
}

Or you can use the ExperimentDirective inline in templates like so:

<div *mExperiment="'my-cool-experiment'; variation: true"
<my-cool-feature></my-cool-feature>
</div>