Past Experiment Learnings
posthog_data/past_experiment_learnings.md
Past Experiment Learnings
Experiment 1: New User NUX Dialog
What was tested: Adding a new user experience (NUX) dialog that appears right after onboarding, guiding the user toward a specific first action. This experiment only ran on web (not native app). A NUX dialog variant was tested against a control group (no dialog).
Source: PostHog Experiment 144921 Time period: Aug 5–21, 2025 (~16 days) Sample size: ~150 users per variant
Results: NUX Dialog vs. Control (No Guidance)
1. User activation — start recording (Funnel) Of all exposed users, what % pressed "start recording" at least once. This is the primary activation metric.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 50.00% | 71 / 142 | — | — |
| NUX Dialog | 62.42% | 98 / 157 | +24.84% | 97.2% |
2. Reached 2+ successful visit notes (Funnel) Users who started recording and went on to generate notes for at least 2 successful patient visits. Measures early engagement depth.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 33.80% | 48 / 142 | — | — |
| NUX Dialog | 42.04% | 66 / 157 | +24.36% | 90.4% |
3. Reached 5+ successful visit notes (Funnel) Users who started recording and went on to generate notes for at least 5 successful patient visits. Measures deeper engagement and habit formation.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 20.42% | 29 / 142 | — | — |
| NUX Dialog | 33.76% | 53 / 157 | +65.30% | 97.6% |
4. Start Recording Count (Mean) Average number of times a user pressed "start recording." Higher means more engagement/repeat usage.
| Variant | Mean | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 3.73 | 530 / 142 | — | — |
| NUX Dialog | 5.34 | 838 / 157 | +42.88% | 95.3% |
5. Started recording success rate (Funnel) Of all exposed users, what % both pressed "start recording" AND the recording actually started (within a 2-minute window). Measures technical recording success across the full user base.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 10.56% | 15 / 142 | — | — |
| NUX Dialog | 17.20% | 27 / 157 | +62.80% | 90.0% |
6. User viewed sample visit (Funnel) Of all exposed users, what % viewed the sample visit note. When a user signs up, the system creates one sample note based on the most common template for the user's specialty. Viewing this sample note is a proxy for early engagement — not as strong as starting a recording, but a signal of user interest that can help activation.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 27.46% | 39 / 142 | — | — |
| NUX Dialog | 44.59% | 70 / 157 | +62.34% | 99.1% |
7. User Subscribed (Funnel) Of all exposed users, what % converted to a paid subscription.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 4.93% | 7 / 142 | — | — |
| NUX Dialog | 8.92% | 14 / 157 | +80.89% | 84.1% |
Key Takeaway
A NUX that guides users toward a specific action right after onboarding significantly improves activation. The NUX dialog outperformed the control (no guidance) across every single metric.
- Activation: 62.42% vs. 50% — a +24.84% lift (97.2% confidence).
- Depth of engagement: Users who received guidance were far more likely to reach 5+ successful visit notes (33.76% vs. 20.42%, +65.30% lift, 97.6% confidence), showing that early activation compounds into sustained usage.
- Repeat usage: Average recordings per user jumped from 3.73 to 5.34 (+42.88%).
- Subscription: Conversion nearly doubled (8.92% vs. 4.93%), confirming the chain: guided first action → activation → deeper engagement → subscription.
- Note: The experiment was likely underpowered (~150 users per variant), so individual metrics did not reach statistical significance. However, the consistent improvement across all 7 metrics is a strong directional signal.
Experiment 2: Quiz Speed Start vs Deep Commitment
See: 001-speed-start-vs-deep-commitment.md
Winner: Speed Start. Shorter 5-question quiz converts 2x better than 13-question deep commitment quiz across all stages. First question is the biggest filter (54.5% drop).
Experiment 3: Speed Start vs ROI Calculator (Results Page)
See: 002-speed-start-vs-roi-calc.md
What was tested: Same speed-start quiz, but different results pages. Control: dark header with "5.0 hours saved" + "+5 patients" framing. Test: clean white ROI layout with motivation-matched headline + testimonial.
Source: PostHog Experiment 364946 Time period: April 7–12, 2026 (~5 days) Sample size: ~2,000 users per variant
Results
| Metric | Control (Speed Start) | Test (ROI Calc) | Delta |
|---|---|---|---|
| Quiz completed | 30.2% | 30.7% | +1.9% |
| Email submitted | 8.3% | 5.8% | -30.4% |
| Signup started | 5.7% | 4.2% | -26.4% |
| Activation (recording) | 1.5% | 1.1% | -27.4% |
Key Takeaway
The results page is the conversion gate, not the quiz. Quiz completion was identical between variants — the entire -30.4% drop in email submissions happened on the results page. The dark header "time saved + patients gained" framing significantly outperforms the clean white ROI layout with motivational headlines.
Caveat resolved by Experiment 4: Experiment 3 confounded content and design changes. The follow-up (see Experiment 4 below) ran with the speed-start visual design held constant and only the content swapped to ROI framing. The drop persisted — content was the dominant cause.
Experiment 4: Speed Start vs ROI Calc v2 (Content-Isolated)
See: 002-speed-start-vs-roi-calc.md for predecessor and 003-speed-start-vs-roi-calc-v2.md for full results.
What was tested: Same speed-start dark-header visual design for both variants. Only the content framing changed — control kept "5 hours / +5 patients" concrete framing; test used ROI-style framing with motivation-matched headlines and personalized config summary.
Source: PostHog Experiment 365716 Time period: April 12, 2026 — present (interim read at ~5,338 exposures) Sample size: ~2,600 users per variant
Results (interim)
| Metric | Control (Speed Start) | Test (ROI Calc v2) | Delta | Win % |
|---|---|---|---|---|
| Quiz completed | 26.81% | 28.76% | +7.25% | 93.5% |
| Email submitted | 6.90% | 5.67% | -17.82% | 2.2% |
| Signup started | 4.57% | 3.54% | -22.63% | 1.6% |
| Account created | 3.95% | 3.16% | -20.11% | 4.1% |
| Activation (start recording) | 1.08% | 0.76% | -29.88% | 7.2% |
Key Takeaway
ROI-style content framing is the cause of the drop, not the design. With the visual layout held constant, ROI content still loses -17.82% on email submission and -29.88% on activation. Concrete "5 hours saved + 5 patients" framing decisively beats ROI-style motivational/personalized framing for cold FB/IG traffic.
Strategic implication: Two experiments now point the same direction. Retire ROI framing as a future challenger. Quiz mechanics (completion is +7.25% directional) are not the bottleneck — the results page signup section is. That is where the next experiment should focus.