Experiment 2: Speed Start vs ROI Calculator
posthog_data/quiz_experiments/002-speed-start-vs-roi-calc.md
Results Page Designs
Losing
Winning
Experiment 2: Speed Start vs ROI Calculator
Source: PostHog Experiment 364946
Date range: April 7–12, 2026
Duration: ~5 days — ended as lost
Statistical significance: No — consistent directional signal across all metrics
Ad context: FB/Insta ads quiz variant experiment
Feature flag: quiz-speed-start-vs-roi-calc
Variants Tested
- Control (Speed Start): The existing speed-start results page — dark header with time-saved stat, light signup section below. Same design that won Experiment 1.
- Test (ROI Calculator): New results page with ROI framing — headline "Be fully present with your clients", larger hours-saved number, personalized config summary ("Configured for BIRP notes, virtual sessions"), testimonial quote, and white/clean design.
Results Page Designs
Control: Speed Start (Winner)
The control uses a dark-to-light split layout:
- Dark header: "Twofold would save you 5.0 hours every week" + "That's +5 patients every week" + "Notes generated in 20 seconds"
- Light section: Email signup form with "Start Free" CTA
- Trust badges: HIPAA Compliant, No credit card, 7-day free trial
- Social proof: "20,000+ clinicians trust Twofold"
Test: ROI Calculator (Loser)
The test uses a clean white layout:
- Motivation-matched headline: "Be fully present with your clients"
- Large stat: "10 hours saved per week"
- Personalization: "Configured for BIRP notes, virtual sessions"
- Email signup form with "Start Free" CTA
- Trust badges: HIPAA Compliant, No credit card, 7-day free trial
- Testimonial: "I went from spending 2 hours every evening on notes to finishing them between sessions. Twofold gets my style perfectly." — Licensed Therapist, Private Practice
Results
Exposure
| Variant | Exposed |
|---|---|
| Control (Speed Start) | 2,036 |
| Test (ROI Calc) | 2,082 |
Primary Metrics
1. Quiz completed (funnel_quiz_completed)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 614 / 2,036 | 30.2% | — |
| Test | 640 / 2,082 | 30.7% | +1.9% |
2. Email submitted (sign_up_email_submitted)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 170 / 2,036 | 8.3% | — |
| Test | 121 / 2,082 | 5.8% | -30.4% |
3. Signup started (sign_up_started)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 117 / 2,036 | 5.7% | — |
| Test | 88 / 2,082 | 4.2% | -26.4% |
4. User activation — start recording
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 31 / 2,036 | 1.5% | — |
| Test | 23 / 2,082 | 1.1% | -27.4% |
Secondary Metrics
5. Quiz started (funnel_quiz_started)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 1,984 / 2,036 | 97.4% | — |
| Test | 2,029 / 2,082 | 97.5% | +0.1% |
6. First question answered
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 709 / 2,036 | 34.8% | — |
| Test | 726 / 2,082 | 34.9% | +0.1% |
Drop-off Analysis
The critical finding: quiz completion is identical between variants (30.2% vs 30.7%). The entire drop happens on the results page — between quiz completion and email submission.
- Quiz start → Quiz complete: no difference (+1.9%)
- Quiz complete → Email submitted: massive drop in test variant (-30.4%)
- This isolates the problem to the results page design/content, not the quiz itself
Key Takeaways
-
The results page is the conversion bottleneck, not the quiz — identical quiz completion rates prove the drop is entirely post-quiz. The ROI calculator results page lost -30.4% of email submissions compared to the speed-start results page.
-
The dark header "time saved" framing outperforms the clean ROI layout — the control's dark-to-light split with concrete "5.0 hours every week" + "+5 patients" framing converts significantly better than the test's white layout with motivation-matched headlines.
-
Content vs design is confounded — this experiment changed both the visual design (dark split vs clean white) AND the content framing (concrete time + patient gains vs motivational headline + testimonial). We cannot determine which factor caused the drop.
-
Revenue impact compounds downstream — the -30.4% email drop cascades to -26.4% signup and -27.4% activation, confirming the results page is the critical conversion gate.
Follow-up
Follow-up experiment: PostHog Experiment 365716 — started April 12, 2026. Full results: 003-speed-start-vs-roi-calc-v2.md.
Same feature flag (quiz-speed-start-vs-roi-calc), restarted to isolate content vs design. Test variant kept the same speed-start visual design (dark header layout) but used ROI calculator content. Hypothesis: if the drop persists, the cause is the content framing; if it recovers, the cause was the design.
Result: drop persists. Content was the cause. Test variant lost -17.82% on email submission, -22.63% on signup started, -29.88% on activation — even with the design held constant. The smaller magnitude vs Experiment 2 (-30.4% email) suggests the white/clean design also hurt, but content was the dominant factor. ROI-style results-page framing is retired as a future challenger.
Winner
Control (Speed Start) — consistent directional winner across all post-quiz metrics. Not statistically significant, but the -30.4% email submission drop is a strong signal.