5.1 Behavioral dynamics: novelty, change aversion, learning curves
At Staff level, your job is to see around corners.
- Novelty effect: numbers jump because the change is shiny, then regress.
- Change aversion: numbers dip because users hate change, then recover.
- Learning effects: adoption is slow, and short tests understate value.
Mitigations:
- Run long enough to see past initial turbulence.
- Analyze cohorts by time since first exposure, not just calendar time.
- Be explicit with stakeholders: "We expect a dip for 1–2 weeks due to change aversion."
Case study: Dark mode rollout
You test a new dark mode in a productivity app.
- Week 1: engagement spikes as curious users toggle the feature.
- Week 3: overall time in app is back to baseline, but night‑time usage is up and complaints about eye strain drop.
By slicing by time of day and time since adoption, you see that novelty washed out, but long‑term behavior changed in the intended segment. Instead of calling it a "failed" experiment, you:
- Roll out dark mode as an opt‑in feature.
- Update success metrics for similar future features to focus on the right cohorts and time windows.
Outcome: you rescue a feature that looked flat on the aggregate metric by telling the right cohort story.
How to talk through this in an interview
- Mid‑level answer: Describe identifying novelty vs. long‑term effects and adjusting the interpretation of the experiment.
- Senior answer: Emphasize cohort slicing, why aggregate averages mislead, and how you changed metric definitions for similar UX changes.
- Staff answer: Frame this as raising the organization’s sophistication: you introduced standard practices for looking at "time since first exposure" and codified expectations around novelty, change aversion, and learning curves in your experimentation playbook.
5.2 Network effects, interference, and marketplaces
If your users interact with each other—or share drivers, inventory, or supply—independence assumptions are broken by default.
- Classic anti‑pattern: user‑level randomization in a social or marketplace product, then surprise when results do not hold at rollout.
5.2.1 Cluster, geo, and switchback designs
Mitigations:
- Cluster or geo randomization when spillovers are meaningful.
- Switchback tests (flip whole systems between control and treatment over time) when you need a marketplace‑level view.
- Be honest about what you cannot measure cleanly.
Case study: Incentives for drivers in a delivery marketplace
Operations wants to test surge‑like incentives for drivers in one city.
- Initial plan: randomize at driver level.
You flag issues:
- Drivers influence each other; incentives for some will change behavior for all.
- Demand and supply are shared; user‑level independence is fiction.
You redesign as geo‑clustered time‑based switchback:
- Entire city is in control or treatment for fixed intervals.
- Compare order completion, ETAs, and driver earnings across windows.
Outcome: results generalize far better, and finance can trust the uplift estimates when deciding whether to fund the program.
How to talk through this in an interview
- Mid‑level answer: Mention that in marketplaces, you used geo‑level or time‑based randomization to avoid spillover effects.
- Senior answer: Clearly explain interference, why driver‑level randomization was invalid, and how the switchback design fixed the identification problem.
- Staff answer: Show that you made this a reusable pattern for marketplace experimentation—sharing templates, educating PMs/ops, and improving the default designs for any change that touches shared supply or demand.
5.2.2 Marketplace health and balanced scorecards
Two‑sided and three‑sided marketplaces (e.g., riders, drivers, restaurants) are where naive experimentation goes to die.
Problems:
- Strong network effects and cross‑side feedback loops.
- Supply‑demand balance that you can easily break with a "small" tweak.
Tools worth knowing:
- Switchback testing: alternate the entire marketplace between control and treatment over time windows to control for temporal noise.
- Geo‑based experiments: randomize at region/city level; analyze at market level.
- Matched markets: pair similar markets and treat only one in each pair.
- Synthetic controls: build a counterfactual from multiple untouched markets when randomization is impossible.
- Incremental rollout: ramp exposure deliberately while watching health metrics across all sides.
Staff role:
- Force the conversation about platform health, not just one metric for one side.
- Make failure modes explicit: "This could improve customer ETA while quietly torching driver retention. Here is how we will detect that early."
Case study: Balancing customer ETAs and driver earnings
A food delivery marketplace wants to test a new dispatch algorithm that promises faster ETAs.
- You design a matched‑market geo experiment: pair similar cities and roll out to one in each pair.
Results after 4 weeks:
- Customer ETAs improve by 7% in treatment cities.
- But driver earnings per hour drop by 5% and driver churn rises, especially in off‑peak hours.
You synthesize:
- Short‑term customer experience is better, but the marketplace is destabilizing on the supply side.
- Long‑term, this likely hurts both sides if driver pool shrinks.
Outcome: instead of full rollout, you push for algorithm changes that explicitly constrain negative impact on driver earnings, then re‑test. The final version trades a smaller ETA gain (3–4%) for stable driver economics—a better equilibrium.
How to talk through this in an interview
- Mid‑level answer: Tell the story of balancing two metrics (ETAs vs. driver earnings) and choosing not to ship the naive "win".
- Senior answer: Explain marketplace dynamics, cross‑side effects, and how you designed the geo experiment plus guardrails to capture those trade‑offs.
- Staff answer: Show how you helped define "marketplace health" as a first‑class concept, ensured future experiments in dispatch/pricing are evaluated on a balanced scorecard, and influenced strategy around sustainable growth.