A / B Testing
Minds uses Growthbook to conduct A/B testing experiments.
Creating an Experiment
Basic Info
To create an experiment, login to Growthbook, and go to the experiments tab. From there you can hit the "New Experiment" button.
Fill out the fields on the first page of the form:
- Name: A concise and relevant name for your experiment.
- Type: Leave this set to "Code" - we have our own wrappers around the SDK.
- Tags: Select any relevant tags.
- Hypothesis: Make your hypothesis? What do we think will happen with the experiment?
- Data Source: Leave set to main warehouse.
When you're done, hit next.
Variations
Here we will be naming and labeling the variations. The "Control" refers to the already exsiting system, and the name can remain as the default 'Control'. For the variation, give it a relevant name and description. Feel free to add more variations if you are testing more than one variant.
Metrics and Targeting
Finally add the relevant fields for Metric and Targeting. At the time of writing it is not possible to add your own Metrics through the Growthbook UI, however in future this will be possible.
When you're done, hit Save. You'll see that this will create your experiment in 'Draft' mode - this will allow others to review your experiment and make changes to it before you make it live. When you are ready, you can start the experiment
Starting an Experiment
When your experiment is ready, you can start it. In the window that opens before starting, you are giving inputs to provide a reason for starting the new phase, time to start, type of phase and importantly, traffic split, which will control what ratio of users see the Control vs your variant.
You should then tie your experiment to a feature flag with the same key, adding an override rule to the feature flag
Developing for an Experiment
Engine
To interact with experiments on the engine side, you will need to use the Experiments Manager.
$experimentsManager = Di::_()->get('Experiments\Manager');
$experimentsManager->setUser($loggedInUser);
// Using isOn for binary experiments:
if ($experimentsManager->isOn('my-cool-experiment')) {
// Cool backend experiment
}
// Using hasVariation, which can be used for ternary experiments:
if ($experimentsManager->hasVariation('my-cool-experiment', true)) {
// Cool backend experiment
}
Front
On the front-end you have a few options - firstly you can use the run
or hasVariation
functions of the ExperimentsService. This can be used as such:
// Using the run function:
public shouldRenderCoolFeature(): boolean {
const output = this.experiments.run('my-cool-experiment');
return output === true;
}
// Using the hasVariation function:
public shouldRenderCoolFeature(): boolean {
return this.experiments.hasVariation('my-cool-experiment', true);
}
Or you can use the ExperimentDirective inline in templates like so:
<div *mExperiment="'my-cool-experiment'; variation: true"
<my-cool-feature></my-cool-feature>
</div>