Past Experiment Learnings
team_feedback/past_experiment_learnings.md
Past Experiment Learnings
Experiment 1: New User NUX Dialog
What was tested: Adding a new user experience (NUX) dialog that appears right after onboarding, guiding the user toward a specific first action. This experiment only ran on web (not native app). A NUX dialog variant was tested against a control group (no dialog).
Source: PostHog Experiment 144921 Time period: Aug 5–21, 2025 (~16 days) Sample size: ~150 users per variant
Results: NUX Dialog vs. Control (No Guidance)
1. User activation — start recording (Funnel) Of all exposed users, what % pressed "start recording" at least once. This is the primary activation metric.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 50.00% | 71 / 142 | — | — |
| NUX Dialog | 62.42% | 98 / 157 | +24.84% | 97.2% |
2. Reached 2+ successful visit notes (Funnel) Users who started recording and went on to generate notes for at least 2 successful patient visits. Measures early engagement depth.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 33.80% | 48 / 142 | — | — |
| NUX Dialog | 42.04% | 66 / 157 | +24.36% | 90.4% |
3. Reached 5+ successful visit notes (Funnel) Users who started recording and went on to generate notes for at least 5 successful patient visits. Measures deeper engagement and habit formation.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 20.42% | 29 / 142 | — | — |
| NUX Dialog | 33.76% | 53 / 157 | +65.30% | 97.6% |
4. Start Recording Count (Mean) Average number of times a user pressed "start recording." Higher means more engagement/repeat usage.
| Variant | Mean | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 3.73 | 530 / 142 | — | — |
| NUX Dialog | 5.34 | 838 / 157 | +42.88% | 95.3% |
5. Started recording success rate (Funnel) Of all exposed users, what % both pressed "start recording" AND the recording actually started (within a 2-minute window). Measures technical recording success across the full user base.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 10.56% | 15 / 142 | — | — |
| NUX Dialog | 17.20% | 27 / 157 | +62.80% | 90.0% |
6. User viewed sample visit (Funnel) Of all exposed users, what % viewed the sample visit note. When a user signs up, the system creates one sample note based on the most common template for the user's specialty. Viewing this sample note is a proxy for early engagement — not as strong as starting a recording, but a signal of user interest that can help activation.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 27.46% | 39 / 142 | — | — |
| NUX Dialog | 44.59% | 70 / 157 | +62.34% | 99.1% |
7. User Subscribed (Funnel) Of all exposed users, what % converted to a paid subscription.
| Variant | Conversion | Raw | Delta | Win % |
|---|---|---|---|---|
| Control | 4.93% | 7 / 142 | — | — |
| NUX Dialog | 8.92% | 14 / 157 | +80.89% | 84.1% |
Key Takeaway
A NUX that guides users toward a specific action right after onboarding significantly improves activation. The NUX dialog outperformed the control (no guidance) across every single metric.
- Activation: 62.42% vs. 50% — a +24.84% lift (97.2% confidence).
- Depth of engagement: Users who received guidance were far more likely to reach 5+ successful visit notes (33.76% vs. 20.42%, +65.30% lift, 97.6% confidence), showing that early activation compounds into sustained usage.
- Repeat usage: Average recordings per user jumped from 3.73 to 5.34 (+42.88%).
- Subscription: Conversion nearly doubled (8.92% vs. 4.93%), confirming the chain: guided first action → activation → deeper engagement → subscription.
- Note: The experiment was likely underpowered (~150 users per variant), so individual metrics did not reach statistical significance. However, the consistent improvement across all 7 metrics is a strong directional signal.