Product & Go-to-Market

Lesson 5 · Product

Experiments & A/B testing

Replace opinion with evidence — and know when the evidence is only noise.

An experiment starts with a hypothesis that could be wrong: “showing pricing on the landing page will raise sign-up conversion”. Split users at random into control (the current version) and treatment (the change), run both at the same time, and compare a metric chosen in advance. Randomization is what lets you attribute the difference to the change rather than to who happened to show up.

Differences also arise by chance. Statistical significance asks how surprising the result would be if the change had no effect, and the p-value is that probability. By convention, below 5% is called significant — which means about one in twenty tests of a change that does nothing will still look like a win. So set the sample size before you start, do not stop the moment the result looks good, and be suspicious of small wins harvested from many tests.

Significance is not importance. A very large test can detect a lift too small to be worth the code. Report the size of the effect with its confidence interval, and watch guardrail metrics — the things that must not get worse, such as latency, refunds or unsubscribes. Not everything should be tested: with little traffic, or for one-way strategic bets, judgment and qualitative research are the right tools.

Picture it

A team tests a new checkout button. After two days the treatment is up 12% and someone wants to ship. The plan said two weeks. By day fourteen the lift is 1.5%, with a confidence interval that spans zero: the early spike was noise plus a weekend. They do not ship, and add “no peeking” to the experiment template.

Key formulas & frameworks

Experiment design
Hypothesis → Random split → Control vs treatment → Pre-chosen metric
Relative lift
(Treatment rate − Control rate) ÷ Control rate
p-value
Probability of a result this extreme if the change did nothing
Guardrail metrics
What must not get worse

Practice

Score: 0 / 5

Question 1

Q1.Control converts at 4.0% and treatment at 4.6%. What is the relative lift, in percent?

Question 2

Q2.A test reports p = 0.03. What does that mean?

Question 3

Q3.Stopping an A/B test as soon as the result looks significant is good practice.

Question 4

Q4.You run 40 tests of changes that truly do nothing, using a 5% significance threshold. How many false wins should you expect?

Question 5

Q5.You are testing more aggressive upgrade prompts. Which is the most useful guardrail metric?

That was the last lessonReady for the certification exam?

Go to the exam