Experiment 3: Speed Start vs ROI Calc v2 (Content-Isolated Follow-up)
posthog_data/quiz_experiments/003-speed-start-vs-roi-calc-v2.md
Experiment 3: Speed Start vs ROI Calc v2 (Content-Isolated Follow-up)
Source: PostHog Experiment 365716
Date range: April 12, 2026 — present (still running at time of read)
Statistical significance: No — but consistent directional signal, win % decisive on the negative side
Ad context: FB/Insta ads quiz variant experiment
Feature flag: quiz-speed-start-vs-roi-calc (re-used from Experiment 2)
Predecessor: 002-speed-start-vs-roi-calc.md
Why This Experiment Exists
Experiment 2 confounded two changes — the test variant changed both visual design (dark split → clean white) AND content framing (concrete time/patient gains → motivational headline + ROI testimonial). It lost -30.4% on email submission, but we couldn't tell whether design or content caused the drop.
Experiment 3 holds visual design constant — both variants use the speed-start dark-header layout. Only the content framing changes. If the test still loses, content is the cause.
Variants Tested
- Control (Speed Start): Same dark-header / light-signup design. Content: "Twofold would save you 5.0 hours every week" + "+5 patients every week" + "Notes generated in 20 seconds".
- Test (ROI Calc, same design): Same dark-header / light-signup design as control. Content: ROI-style framing — larger hours-saved number with personalized config summary and motivation-matched headlines, but rendered in the speed-start visual shell.
Results (interim read, experiment still running)
Exposure
| Variant | Exposed | Share |
|---|---|---|
| Control | 2,580 | 48.4% |
| Test | 2,630 | 49.3% |
| $multiple | ~128 | 2.4% |
| Total | 5,338 | 100% |
Primary Metrics
1. Quiz completed (funnel_quiz_completed)
| Variant | Count | Rate | Delta | Win % |
|---|---|---|---|---|
| Control | 692 / 2,580 | 26.81% | — | — |
| Test | 756 / 2,630 | 28.76% | +7.25% | 93.5% |
2. Email submitted (sign_up_email_submitted)
| Variant | Count | Rate | Delta | Win % |
|---|---|---|---|---|
| Control | 178 / 2,580 | 6.90% | — | — |
| Test | 149 / 2,630 | 5.67% | -17.82% | 2.2% |
3. Signup started (sign_up_started)
| Variant | Count | Rate | Delta | Win % |
|---|---|---|---|---|
| Control | 118 / 2,580 | 4.57% | — | — |
| Test | 93 / 2,630 | 3.54% | -22.63% | 1.6% |
4. User activation — start recording
| Variant | Count | Rate | Delta | Win % |
|---|---|---|---|---|
| Control | 28 / 2,580 | 1.08% | — | — |
| Test | 20 / 2,630 | 0.76% | -29.88% | 7.2% |
5. Account created (clerk_user_created)
| Variant | Count | Rate | Delta | Win % |
|---|---|---|---|---|
| Control | 102 / 2,580 | 3.95% | — | — |
| Test | 83 / 2,630 | 3.16% | -20.11% | 4.1% |
Drop-off Analysis
The pattern from Experiment 2 reproduces almost exactly, even with the design controlled:
- Quiz completed: test slightly better (+7.25%, 93.5% win) — so users engage with the quiz fine
- Email submitted: test loses -17.82% (win % 2.2%)
- Signup started: test loses -22.63% (win % 1.6%)
- Activation: test loses -29.88% (win % 7.2%)
- Account created: test loses -20.11% (win % 4.1%)
The funnel breaks specifically at the moment users see the results page content. Quiz mechanics are not the issue.
Key Takeaways
-
Content framing was the cause, not the design. With the design held constant, ROI-style content still loses -17.8% on email submission. The "5 hours saved + 5 patients gained" concrete framing is the active ingredient — not the dark-header visual treatment.
-
The quiz itself is not the conversion problem on this surface. Quiz completion rate is directionally higher in the test (+7.25%, 93.5% win). More users finish the quiz and still convert worse on the results page. This isolates the bottleneck to the post-quiz / signup surface.
-
ROI framing is retired as a future challenger. Two consecutive experiments (002 with both changed, 003 with content isolated) point the same direction. Stop testing ROI-style results-page content variants.
-
The next experiment should target the results page signup section directly — the mechanism that converts a quiz-completer into an email submission. Quiz length, quiz content, and dashboard-style results are not where the leverage is.
-
Cascade is consistent. Email -17.8% → signup-started -22.6% → account-created -20.1% → activation -29.9%. Each downstream step amplifies the upstream drop, so wins on the email-submission surface compound through the funnel.
Winner
Control (Speed Start) — directional winner across all four downstream metrics with decisive win percentages on the negative side (test win % between 1.6% and 7.2% on every conversion metric). Not statistically significant in the strict sense, but the signal is consistent across two experiments and four metrics within this one. Treat as confirmed.
What This Confirms vs. Experiment 2
| Hypothesis from Exp 2 follow-up | Result |
|---|---|
| If drop persists with same design → content was cause | Confirmed. Drop persists (-17.82% email vs Exp 2's -30.4%). |
| If drop disappears with same design → design was cause | Rejected. Drop did not disappear. |
The drop in Experiment 3 (-17.8% email) is smaller than Experiment 2 (-30.4% email), suggesting the white/clean design also hurt — but content was the dominant factor.