Past Experiment Learnings
posthog_data/past_experiment_learnings.md
Past Experiment Learnings
Experiment 1: New User NUX Dialog
What was tested: Adding a new user experience (NUX) dialog that appears right after onboarding, guiding the user toward a specific first action. This experiment only ran on web (not native app). A NUX dialog variant was tested against a control group (no dialog).
Source: PostHog Experiment 144921 Time period: Aug 5–21, 2025 (~16 days) Sample size: ~150 users per variant
Results: NUX Dialog vs. Control (No Guidance)
1. User activation — start recording (Funnel) Of all exposed users, what % pressed "start recording" at least once. This is the primary activation metric.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 50.00% | 71 / 142 | — | — |
| NUX Dialog | 62.42% | 98 / 157 | +24.84% | 97.2% |
2. Reached 2+ successful visit notes (Funnel) Users who started recording and went on to generate notes for at least 2 successful patient visits. Measures early engagement depth.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 33.80% | 48 / 142 | — | — |
| NUX Dialog | 42.04% | 66 / 157 | +24.36% | 90.4% |
3. Reached 5+ successful visit notes (Funnel) Users who started recording and went on to generate notes for at least 5 successful patient visits. Measures deeper engagement and habit formation.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 20.42% | 29 / 142 | — | — |
| NUX Dialog | 33.76% | 53 / 157 | +65.30% | 97.6% |
4. Start Recording Count (Mean) Average number of times a user pressed "start recording." Higher means more engagement/repeat usage.
| Variant | Mean | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 3.73 | 530 / 142 | — | — |
| NUX Dialog | 5.34 | 838 / 157 | +42.88% | 95.3% |
5. Started recording success rate (Funnel) Of all exposed users, what % both pressed "start recording" AND the recording actually started (within a 2-minute window). Measures technical recording success across the full user base.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 10.56% | 15 / 142 | — | — |
| NUX Dialog | 17.20% | 27 / 157 | +62.80% | 90.0% |
6. User viewed sample visit (Funnel) Of all exposed users, what % viewed the sample visit note. When a user signs up, the system creates one sample note based on the most common template for the user's specialty. Viewing this sample note is a proxy for early engagement — not as strong as starting a recording, but a signal of user interest that can help activation.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 27.46% | 39 / 142 | — | — |
| NUX Dialog | 44.59% | 70 / 157 | +62.34% | 99.1% |
7. User Subscribed (Funnel) Of all exposed users, what % converted to a paid subscription.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 4.93% | 7 / 142 | — | — |
| NUX Dialog | 8.92% | 14 / 157 | +80.89% | 84.1% |
Key Takeaway
A NUX that guides users toward a specific action right after onboarding significantly improves activation. The NUX dialog outperformed the control (no guidance) across every single metric.
- Activation: 62.42% vs. 50% — a +24.84% lift (97.2% confidence).
- Depth of engagement: Users who received guidance were far more likely to reach 5+ successful visit notes (33.76% vs. 20.42%, +65.30% lift, 97.6% confidence), showing that early activation compounds into sustained usage.
- Repeat usage: Average recordings per user jumped from 3.73 to 5.34 (+42.88%).
- Subscription: Conversion nearly doubled (8.92% vs. 4.93%), confirming the chain: guided first action → activation → deeper engagement → subscription.
- Note: The experiment was likely underpowered (~150 users per variant), so individual metrics did not reach statistical significance. However, the consistent improvement across all 7 metrics is a strong directional signal.
Experiment 2: Quiz Speed Start vs Deep Commitment
See: 001-speed-start-vs-deep-commitment.md
Winner: Speed Start. Shorter 5-question quiz converts 2x better than 13-question deep commitment quiz across all stages. First question is the biggest filter (54.5% drop).
Experiment 3: Speed Start vs ROI Calculator (Results Page)
See: 002-speed-start-vs-roi-calc.md
What was tested: Same speed-start quiz, but different results pages. Control: dark header with "5.0 hours saved" + "+5 patients" framing. Test: clean white ROI layout with motivation-matched headline + testimonial.
Source: PostHog Experiment 364946 Time period: April 7–12, 2026 (~5 days) Sample size: ~2,000 users per variant
Results
| Metric | Control (Speed Start) | Test (ROI Calc) | Delta |
|---|---|---|---|
| Quiz completed | 30.2% | 30.7% | +1.9% |
| Email submitted | 8.3% | 5.8% | -30.4% |
| Signup started | 5.7% | 4.2% | -26.4% |
| Activation (recording) | 1.5% | 1.1% | -27.4% |
Key Takeaway
The results page is the conversion gate, not the quiz. Quiz completion was identical between variants — the entire -30.4% drop in email submissions happened on the results page. The dark header "time saved + patients gained" framing significantly outperforms the clean white ROI layout with motivational headlines.
Caveat: This experiment confounded content and design changes. Follow-up Experiment 365716 is running with the speed-start visual design but ROI content to isolate which factor caused the drop.