Run & Monitor
Monitor your experiment's performance in real time.
Run & Monitor
Once your experiment is configured, this page explains how to start it, monitor progress, and manage it while it runs.
Starting an experiment
- Go to Experiments in the sidebar.
- Open your draft experiment.
- Click Start.
From this point, eligible users who trigger the variant flows will be assigned to a variant based on your traffic split. Assignment is persistent -- a user always sees the same variant throughout the experiment.
Experiments respect each variant flow's existing trigger conditions. Only users who match a flow's targeting rules are eligible for that variant.
The results dashboard
The experiment detail page shows real-time results that auto-refresh every 30 seconds. The dashboard includes:
Metrics summary
A view of each variant's performance on the primary metric, including:
- The variant name and its assigned flow.
- The number of users exposed to the variant.
- The primary metric value.
- Lift over control (percentage difference compared to the control variant).
- Statistical confidence for the current result.
Sample size progress
Setgreet indicates whether the minimum sample size has been reached for each variant. Until the minimum is met, results are still considered preliminary.
Winner recommendation
Once the primary metric shows a clear leader and the sample size is adequate, a Winner badge is displayed on the leading variant along with the lift over control.
Pausing an experiment
If you need to temporarily halt an experiment:
- Click Pause on the experiment detail page.
- The experiment enters the Paused state.
While paused:
- No new users are assigned to variants.
- All collected data is preserved.
You can resume a paused experiment by clicking Resume. New users will begin being assigned again.
Pausing and resuming an experiment can introduce bias if the user population changes between the pause and resume periods. Avoid pausing unless necessary.
Completing an experiment
When you have collected enough data and are ready to finalize the results, click Complete. This:
- Stops assigning new users to variants.
- Moves the experiment to the Completed state.
- Optionally records the winning variant for reference.
Once completed, proceed to Interpret Results to decide what to ship.
How long should you run an experiment?
There is no universal answer, but consider:
- Sample size -- wait until the minimum sample size is met for every variant.
- Effect size -- if you expect a large difference between variants (for example, testing completely different approaches), you need fewer users. Subtle differences need larger samples.
- Stability -- metrics often fluctuate early on. Wait until the numbers stabilize before drawing conclusions.