Experiment 1: Speed Start vs Deep Commitment
posthog_data/quiz_experiments/001-speed-start-vs-deep-commitment.md
Experiment 1: Speed Start vs Deep Commitment
Source: PostHog Experiment 360320 Date range: March 2026 (a few days) Duration: Stopped early — statistically significant results Statistical significance: Yes Ad context: Ads targeted time savings and burnout. No pain-point segmentation. Deployed code:
- Speed Start:
twofold/apps/quiz-landing-nextjs/src/variants/speed-start/ - Deep Commitment:
twofold/apps/quiz-landing-nextjs/src/variants/deep-commitment/
Variants Tested
- V1 (Speed Start): 5-question quiz, ~90 seconds, minimum friction. Strategic bet: the fastest path from FB ad click to first recording converts the most cold mobile traffic.
- V2 (Deep Commitment): 13-question phased quiz across 4 phases with interstitials, 4-5 minutes. Strategic bet: higher investment = higher activation, following Noom's long-quiz model.
Deployed Quiz Content
V1: Speed Start
5 questions, linear flow:
| # | Question | Options |
|---|---|---|
| 1 | "What's your primary role?" | Therapist, Psychiatrist, Psychologist, Social Worker, Primary Care, Other |
| 2 | "How do most of your sessions happen?" | In person, Virtual, Both |
| 3 | "When do you usually finish your notes?" | Between sessions, End of day, Evenings & weekends, I'm behind on notes |
| 4 | "What would help most right now?" | Finish notes faster, Stay present in sessions, Reduce compliance stress, Get my evenings back |
| 5 | "Which note format do you use?" | SOAP, DAP, BIRP, Progress Notes, Not sure |
Flow: Q1 → Q2 → Q3 → Q4 → Q5 → Loading screen → Results page → Signup form
Loading screen: "Setting up your workspace" with checklist:
- "Loading [Role] templates"
- "Setting note format to [Format]"
- "Configuring your workspace"
Results page:
- Hero stat: hours saved per week (calculated from note timing answer: Between sessions = 4.5h, End of day = 6h, Evenings & weekends = 10h, Behind = "Today / clear your backlog")
- Motivation-matched headline (mapped from Q4 answer):
- "Finish notes faster" → "Your notes, done before the next session"
- "Stay present in sessions" → "Be fully present with your clients"
- "Reduce compliance stress" → "Compliant notes without the stress"
- "Get my evenings back" → "Your evenings are yours again"
- Description: "Twofold listens to your sessions and writes your [Format] notes automatically."
- Personalization summary: role, template (format — preloaded), session type
- Testimonial: "I went from spending 2 hours every evening on notes to finishing them between sessions. Twofold gets my style perfectly." — Licensed Therapist, Private Practice
- Social proof: 4.8/5 rating, 20,000+ clinicians, HIPAA
- CTA: "Start Free"
Signup form:
- Headline: "Create your account"
- Description: "Your [Format] setup for [Role] sessions is ready."
- Fields: Email ("you@practice.com"), Password ("Create a password")
- CTA: "Start Free"
- Legal text: "By signing up you agree to the Terms of Use and Privacy Policy"
- Social proof: HIPAA compliant, No credit card
V2: Deep Commitment
13 questions across 4 phases with interstitials and loading screens:
Phase 1 — "About Your Practice" (Q1-Q3):
| # | Question | Options |
|---|---|---|
| 1 | "What's your primary clinical role?" | Therapist, Psychiatrist, Psychologist, Social Worker, Primary Care, Other |
| 2 | "What's your practice size?" | Just me (solo practice), 2-9 clinicians, 10-49 clinicians, 50+ clinicians |
| 3 | "How do most of your sessions happen?" | In person, Virtual, Both |
→ Interstitial 1: "78% of [role] providers report documentation burnout" / "Administrative burden is the #1 driver of clinician burnout. You're not alone — and there's a better way."
Phase 2 — "Your Documentation" (Q4-Q7):
| # | Question | Options |
|---|---|---|
| 4 | "When do you usually finish your notes?" | Between sessions, End of day, Evenings & weekends, I'm behind on notes |
| 5 | "How many hours per week on documentation?" | Less than 5, 5-10 hours, 10-15 hours, More than 15 |
| 6 | "What frustrates you most about documentation?" | It takes too long, It's repetitive, It cuts into personal time, Hard to capture everything |
| 7 | "Have you tried other AI documentation tools?" | Yes liked some, Yes didn't work for me, No, Not sure what's out there |
→ Interstitial 2: "You could save ~[X] hours every week" / "That's ~[X*4] hours a month you could spend on what matters — your clients, your family, yourself." (X = 80% of reported doc hours)
Phase 3 — "Your Workflow" (Q8-Q10):
| # | Question | Options |
|---|---|---|
| 8 | "What note format do you use most?" | SOAP, DAP, BIRP, Progress Notes, Intake Notes, Not sure |
| 9 | "How do you currently create your notes?" | Typing from memory, Copy-paste templates, Dictation/voice, EHR built-in tools |
| 10 | "How would you prefer to generate notes?" | AI listens and writes, Dictate after session, Smart templates, Open to anything faster |
→ Loading screen 1: "Matching your template...", "Configuring for your sessions...", "Almost ready..."
Phase 4 — "Your Goals" (Q11-Q13):
| # | Question | Options |
|---|---|---|
| 11 | "What would better documentation mean for your life?" | More time with family, More presence with clients, Less burnout, Grow my practice |
| 12 | "How would you describe your ideal workflow?" | Fully automated, AI drafts I review, I write AI assists, Not sure yet |
| 13 | "What matters most in a documentation tool?" | Accuracy, Speed, Compliance, Ease of use |
→ Loading screen 2: "Analyzing your documentation profile...", "Calculating time savings...", "Finalizing your personalized setup..."
Results page: 4-card personalized dashboard:
- Your Practice — "[Role] · [Practice Size] · [Modality]"
- Your Challenge — "You spend [doc hours]/week on documentation. You finish notes [timing]. Your biggest frustration: [frustration]."
- Your Twofold Setup — "[Format] template loaded. [Modality] mode configured. Primary workflow: [preferred method]. Optimized for [priority]."
- Your Potential Impact (highlighted gradient card) — "~[X] hrs/week saved. That means: [aspiration]."
- Headline: "Here's your personalized documentation plan"
- Description: "Twofold listens to your sessions and writes your notes automatically — no typing, no templates, no after-hours charting."
- Frustration-matched headline (mapped from Q6):
- "It takes too long" → "Your notes, done before the next session"
- "It's repetitive" → "No more copy-paste. Every note written for you"
- "It cuts into personal time" → "Your evenings are yours again"
- "Hard to capture everything" → "Every detail captured, nothing missed"
- Same testimonial and social proof as V1
- CTA: "Start Free"
Key structural differences from V1:
- 8 additional questions (practice size, doc hours, frustration, tried tools, current method, preferred method, aspiration, ideal workflow, priority)
- 2 interstitials between phases (motivational/value screens)
- 2 loading screens (vs 1 in V1)
- Richer results page (4-card dashboard vs single hero stat)
- Progress bar: continuous percentage-based (vs dot-style step indicators in V1)
Results
Required Metrics
1. Quiz start → Quiz complete (reach signup/results page)
| Variant | Conversion |
|---|---|
| V1: Speed Start | 43.0% |
| V2: Deep Commitment | 26.4% |
2. Quiz complete → Signup complete
| Variant | Conversion |
|---|---|
| V1: Speed Start | 13.45% |
| V2: Deep Commitment | 11.31% |
3. Signup → First recording (activation)
| Variant | Signed Up | First Recording | Rate |
|---|---|---|---|
| V1: Speed Start | 885 | 53 | 5.99% |
| V2: Deep Commitment | 849 | 25 | 2.94% |
Drop-off Analysis
V1: Speed Start
- Major drop at Q1: 1,062 saw Q1 → 483 answered Q1 (45.48% survival)
- After Q1: small 2-3% drops per question — very healthy retention once committed
- The first question is the single largest filter in the entire funnel
V2: Deep Commitment
- Q1 → Q4: only 83.2% survived (expected ~90%+ if drops were consistent 2-3% each)
- A problematic question between Q1 and Q4 is causing excess drop-off beyond normal quiz fatigue
- Note: Q2 ("What's your practice size?") and Q3 ("How do most of your sessions happen?") + Interstitial 1 sit between Q1 and Q4. One or more of these is likely the friction point.
Key Takeaways
- Shorter quizzes convert significantly better — V1's 43% quiz completion vs V2's 26.4% validates that minimum friction wins for cold FB/IG ad traffic
- First question is the biggest filter — 54.52% of V1 users who see Q1 drop before answering. This is the single largest optimization opportunity regardless of which variation wins
- Post-quiz signup conversion is similar — 13.45% vs 11.31% suggests quiz structure matters more than quiz length for signup conversion
- Activation advantage compounds — V1's quiz-to-activation pipeline produced 2x the activated users (53 vs 25)
- V2 has mid-quiz friction — The sub-83.2% Q1→Q4 survival suggests specific question(s) in the early phase are causing abandonment. The interstitial or the practice-size question (not present in V1) may be the cause.
Winner
V1 (Speed Start) — statistically significant winner across all three stages (quiz completion, signup, activation).