Lesson 5 · Product
Experiments & A/B testing
Replace opinion with evidence — and know when the evidence is only noise.
An experiment starts with a hypothesis that could be wrong: “showing pricing on the landing page will raise sign-up conversion”. Split users at random into control (the current version) and treatment (the change), run both at the same time, and compare a metric chosen in advance. Randomization is what lets you attribute the difference to the change rather than to who happened to show up.
Differences also arise by chance. Statistical significance asks how surprising the result would be if the change had no effect, and the p-value is that probability. By convention, below 5% is called significant — which means about one in twenty tests of a change that does nothing will still look like a win. So set the sample size before you start, do not stop the moment the result looks good, and be suspicious of small wins harvested from many tests.
Significance is not importance. A very large test can detect a lift too small to be worth the code. Report the size of the effect with its confidence interval, and watch guardrail metrics — the things that must not get worse, such as latency, refunds or unsubscribes. Not everything should be tested: with little traffic, or for one-way strategic bets, judgment and qualitative research are the right tools.
Picture it
A team tests a new checkout button. After two days the treatment is up 12% and someone wants to ship. The plan said two weeks. By day fourteen the lift is 1.5%, with a confidence interval that spans zero: the early spike was noise plus a weekend. They do not ship, and add “no peeking” to the experiment template.
Key formulas & frameworks
- Experiment design
- Hypothesis → Random split → Control vs treatment → Pre-chosen metric
- Relative lift
- (Treatment rate − Control rate) ÷ Control rate
- p-value
- Probability of a result this extreme if the change did nothing
- Guardrail metrics
- What must not get worse
Practice
Score: 0 / 5
That was the last lessonReady for the certification exam?
Go to the exam