Who This Helps
You're a Product Manager with a backlog full of experiment ideas. Every option looks good on paper. But you only have time for one. You need a way to pick the move that actually moves the needle.
This is for you if you've ever felt like your team is running experiments just to run them. Or if you've shipped a feature and realized later it didn't change anything.
Mini Case
Meet Priya. She's a PM at a SaaS company. Her team has three experiment ideas for next sprint:
- Idea A: Simplify the onboarding flow
- Idea B: Add a new dashboard widget
- Idea C: Send a re-engagement email
Priya's activation rate is stuck at 22%. She knows that if she can get it to 30%, retention jumps by 12%. But which experiment gets her there?
She uses the Product Metrics Basics program to define her activation metric: one action (complete first report) within one window (7 days). Then she checks her segment snapshot. Turns out, users who complete the first report in 3 days have a 40% higher retention rate.
Now the choice is clear: Idea A (simplify onboarding) directly impacts that 3-day window. She prioritizes it. The experiment shows a 15% lift in activation.
Do This Now (5 Steps)
- Define your activation metric. Pick one action and one time window. For example: "Complete first report within 7 days." This is your decision anchor.
- Check your current activation rate. Look at the last 30 days. If it's below 30%, that's your focus area.
- Find the segment that breaks. Use a segment snapshot. Compare activation rates by user type, signup source, or plan. Find the group with the biggest drop-off.
- List your experiment ideas. Write down 3-5 options. For each one, ask: "Does this directly improve the activation metric for the broken segment?"
- Pick the one with the highest potential impact. If an idea doesn't touch the metric, kill it. If it does, run it first.
Avoid These Traps
- Picking based on gut. Your intuition is useful, but activation data is more reliable. Use the numbers.
- Running too many experiments at once. You'll split your team's focus and get muddy results. Pick one.
- Ignoring the time window. If your activation window is 7 days, don't optimize for day 1. That's a different metric.
- Forgetting guardrails. A North Star metric keeps you honest. But also set guardrails (like support ticket volume) so you don't break something else.
- Not checking the segment. A 22% overall rate might hide a 50% rate for one segment and 10% for another. Always slice.
Your Win by Friday
By Friday, you will have:
- One clear activation metric (action + window)
- One segment snapshot showing where activation breaks
- One experiment prioritized based on that data
That's it. No more guessing. No more debating. Just a decision backed by a metric you trust.
And hey, if you nail this, you might even free up time to grab coffee before the next standup.