How to A/B test your app's onboarding flow

To A/B test onboarding well you need three things: variants that differ by one meaningful variable, a success metric further downstream than completion, and infrastructure that lets you ship the next test without an app release. Most teams fail on the third, which is why most onboarding flows have never been tested at all.

Why mobile A/B testing is structurally hard

On the web, an experiment is a deploy. In a mobile app, a hardcoded onboarding variant is a build, a review, a phased rollout, and a wait for the installed base to update, per variant. By the time results arrive, the team has moved on. This is not a discipline problem; it is a tooling problem. Testing velocity is worth engineering for: an analysis published by Adapty in 2026 found apps running 50-plus experiments had a median revenue 18.7x that of apps that ran a single one. That is correlation, not causation (teams that test a lot differ in many ways), but the direction matches what every high-performing subscription team reports about its own process.

What to test, in order of leverage

1. Paywall placement and framing. End-of-onboarding vs feature-gate, trial-led vs price-led. The highest-impact structural variable; our paywall placement guide covers the baselines.
2. Flow length and question order. Which personalization questions stay, which go, and what order builds the strongest commitment loop.
3. Value framing. The first screen's promise: outcome-led vs feature-led, social proof vs plan preview.
4. Permission prompt timing. Where the notification primer sits, and what it says.
5. Copy and design. Real, but last. Button colors are the homeopathy of onboarding optimization.

One variable per test. A variant that changes the paywall, the length, and the copy at once can only tell you that something worked, which teaches you nothing reusable.

Pick the right success metric

Completion rate is the seductive wrong answer. Cutting screens nearly always raises completion; whether it helps the business depends on what those screens did. The variant that wins on completion can lose on trial starts, and the variant that wins on trial starts can lose on Day 7 retention if it over-promises. Define the primary metric one step past the flow: activation (first core action), trial start, or early retention. Use completion as a diagnostic, not a verdict.

Respect the sample-size math

Onboarding experiments are gated by new installs per day, and subscription events are sparse. A few honest rules:

• Size the test before you start: baseline rate, minimum effect you care about, and the installs per week you actually get. Small apps often need weeks per test; that is normal, plan around it.
• Do not peek and stop at the first significant-looking gap. Early leads flip constantly; commit to the planned sample.
• Chase effects worth detecting. With modest traffic you cannot resolve a 2% relative lift, and you do not need to: structural changes (placement, length) move numbers by tens of percent when they work.
• Segment after, not during. Slicing mid-test by platform, source, and geography until something is significant is how teams fool themselves.

Run the loop with Setgreet

This whole discipline collapses without cheap iteration. Setgreet runs A/B experiments on native flows: define variants in the editor, split traffic, and read results against flow-level analytics, with no release between test and next test. Experiments are part of the paid plans, and every plan, including free, keeps full analytics, so you can see your funnel before you ever pay.

A worked example

Say you run a meditation app doing 700 installs a week, onboarding ends in a trial-led paywall, and 8% of installs start a trial. Hypothesis: moving the paywall from screen 6 to screen 9, after the plan-preview moment, will lift trial starts. You care about a lift of at least 20% relative (8% to 9.6%); anything smaller does not justify the flow being three screens longer.

1. Sample-size math at those numbers says roughly 5,500 users per arm for a standard test, so plan for about eight weeks with a 50/50 split, and pre-commit to it in writing.
2. Primary metric: trial starts per install. Guardrails: onboarding completion and Day 7 retention, watched for damage rather than victory.
3. At week 3 variant B leads by 31%. This is exactly the moment the discipline exists for: keep running. Early leads of that size regularly decay.
4. At week 8: B shows a 24% relative lift on trial starts, completion dropped two points, Day 7 retention is flat. Ship B.
5. The next test starts from B as the new baseline: same slot, trial-led vs price-led framing.

Two things to notice. The test was worth eight weeks because placement is a structural variable with a plausible 20%-plus effect; the same eight weeks on a button-color test would have been thrown away. And the guardrail metrics did real work: a completion drop with flat retention says the extra screens filtered browsers, not buyers, which is exactly the trade you wanted.

Frequently asked questions

How long should an onboarding A/B test run?

Until the pre-committed sample size is reached, and at least one full week to wash out day-of-week effects. For most apps that means two to four weeks per test; below roughly a thousand installs a week, favor bigger structural changes that need smaller samples to detect.

Can I test onboarding without a testing platform?

You can ship variant A, then variant B, and compare cohorts. This before-and-after method is vulnerable to seasonality and acquisition-mix shifts, but it is far better than nothing, and it becomes reasonably trustworthy when the effect is large and the cohorts are adjacent.

What win rate should I expect from onboarding tests?

Expect most tests to lose or tie; published experiment win rates across the industry are well under half. The economics work anyway, because losers are cheap to kill and winners compound permanently. A team that ships one real winner a quarter transforms its funnel in a year.