1.1 Role of experimentation at Staff level
At senior levels, being "good at experiments" does not mean:
- Knowing every statistical test by name
- Arguing about p‑values on Slack
- Running more tests than anyone else
It means you reliably:
- Frame the right questions and kill the wrong ones early
- Choose the right method (including deciding not to experiment)
- Connect results to strategy, not just metrics
- Build systems so others can run high‑quality experiments without you in the room
A Staff DS is the person who can say "No, this experiment is a waste of traffic" and be right.
Case study: Saying "no" to a flashy homepage test
A PM wants to test a flashy new homepage hero that might increase clicks but does not connect to any downstream value. You push back:
- The North Star is weekly retained buyers, not homepage CTR.
- There is no credible mechanism from a cosmetic hero change to long‑term retention.
- The required MDE on retained buyers would make the test huge and low‑value.
Outcome: you decline the experiment, redirect traffic budget to testing improvements in search relevance, and later ship a search change that measurably lifts orders. Your impact came from prioritization, not running more tests.
How to talk through this in an interview
- Mid‑level answer: Focus on explaining why the homepage hero is not clearly tied to core metrics and suggest measuring downstream conversion instead. Emphasize that you would still be open to running the test if you can tie it to a better metric.
- Senior answer: Explicitly walk through the metric stack (North Star vs. CTR), quantify the MDE problem, and show how you redirected experimentation capacity toward a higher‑leverage search experiment.
- Staff answer: Frame this as a portfolio and governance problem: experimentation is a scarce resource, so you created a lightweight review bar for experiments, pushed for hypotheses grounded in mechanisms, and used this case to shape how PMs propose tests going forward.