Experiment 2: Speed Start vs ROI Calculator
posthog_data/quiz_experiments/002-speed-start-vs-roi-calc.md
Results Page Designs
Losing
Winning
Experiment 2: Speed Start vs ROI Calculator
Source: PostHog Experiment 364946
Date range: April 7–12, 2026
Duration: ~5 days — ended as lost
Statistical significance: No — consistent directional signal across all metrics
Ad context: FB/Insta ads quiz variant experiment
Feature flag: quiz-speed-start-vs-roi-calc
Variants Tested
- Control (Speed Start): The existing speed-start results page — dark header with time-saved stat, light signup section below. Same design that won Experiment 1.
- Test (ROI Calculator): New results page with ROI framing — headline "Be fully present with your clients", larger hours-saved number, personalized config summary ("Configured for BIRP notes, virtual sessions"), testimonial quote, and white/clean design.
Results Page Designs
Control: Speed Start (Winner)
The control uses a dark-to-light split layout:
- Dark header: "Twofold would save you 5.0 hours every week" + "That's +5 patients every week" + "Notes generated in 20 seconds"
- Light section: Email signup form with "Start Free" CTA
- Trust badges: HIPAA Compliant, No credit card, 7-day free trial
- Social proof: "20,000+ clinicians trust Twofold"
Test: ROI Calculator (Loser)
The test uses a clean white layout:
- Motivation-matched headline: "Be fully present with your clients"
- Large stat: "10 hours saved per week"
- Personalization: "Configured for BIRP notes, virtual sessions"
- Email signup form with "Start Free" CTA
- Trust badges: HIPAA Compliant, No credit card, 7-day free trial
- Testimonial: "I went from spending 2 hours every evening on notes to finishing them between sessions. Twofold gets my style perfectly." — Licensed Therapist, Private Practice
Results
Exposure
| Variant | Exposed |
|---|---|
| Control (Speed Start) | 2,036 |
| Test (ROI Calc) | 2,082 |
Primary Metrics
1. Quiz completed (funnel_quiz_completed)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 614 / 2,036 | 30.2% | — |
| Test | 640 / 2,082 | 30.7% | +1.9% |
2. Email submitted (sign_up_email_submitted)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 170 / 2,036 | 8.3% | — |
| Test | 121 / 2,082 | 5.8% | -30.4% |
3. Signup started (sign_up_started)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 117 / 2,036 | 5.7% | — |
| Test | 88 / 2,082 | 4.2% | -26.4% |
4. User activation — start recording
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 31 / 2,036 | 1.5% | — |
| Test | 23 / 2,082 | 1.1% | -27.4% |
Secondary Metrics
5. Quiz started (funnel_quiz_started)
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 1,984 / 2,036 | 97.4% | — |
| Test | 2,029 / 2,082 | 97.5% | +0.1% |
6. First question answered
| Variant | Count | Rate | Delta |
|---|---|---|---|
| Control | 709 / 2,036 | 34.8% | — |
| Test | 726 / 2,082 | 34.9% | +0.1% |
Drop-off Analysis
The critical finding: quiz completion is identical between variants (30.2% vs 30.7%). The entire drop happens on the results page — between quiz completion and email submission.
- Quiz start → Quiz complete: no difference (+1.9%)
- Quiz complete → Email submitted: massive drop in test variant (-30.4%)
- This isolates the problem to the results page design/content, not the quiz itself
Key Takeaways
-
The results page is the conversion bottleneck, not the quiz — identical quiz completion rates prove the drop is entirely post-quiz. The ROI calculator results page lost -30.4% of email submissions compared to the speed-start results page.
-
The dark header "time saved" framing outperforms the clean ROI layout — the control's dark-to-light split with concrete "5.0 hours every week" + "+5 patients" framing converts significantly better than the test's white layout with motivation-matched headlines.
-
Content vs design is confounded — this experiment changed both the visual design (dark split vs clean white) AND the content framing (concrete time + patient gains vs motivational headline + testimonial). We cannot determine which factor caused the drop.
-
Revenue impact compounds downstream — the -30.4% email drop cascades to -26.4% signup and -27.4% activation, confirming the results page is the critical conversion gate.
Follow-up
Follow-up experiment: PostHog Experiment 365716 — started April 12, 2026.
Same feature flag (quiz-speed-start-vs-roi-calc), restarted to isolate content vs design. The test variant now uses the same speed-start visual design (dark header layout) but with ROI calculator content. If the drop persists, the cause is the content framing. If it recovers, the cause was the design.
Winner
Control (Speed Start) — consistent directional winner across all post-quiz metrics. Not statistically significant, but the -30.4% email submission drop is a strong signal.