"Conversions went up after we shipped it" and "the change caused conversions to go up" are different claims, and confusing them is how teams accumulate a folder of best practices that don't reproduce.
Between the two sits everything else that changed that week: a campaign, a season, a competitor, a payday, a broken tracking script.
Measure the step, not the total
The first and largest improvement to your evidence costs nothing: watch the transition you tried to move.
If you removed a field from your checkout form, look at checkout-started → completed. Not sitewide conversion, which is diluted by every visitor who never reached checkout, and contaminated by everything else that happened.
A change to one step should show up at that step. If it moves your total but not the step, you didn't cause it.
Change one thing
If you shipped three changes, you have one number and three candidate explanations. There is no analysis that recovers the answer afterward.
This is unglamorous and it is the difference between a team whose knowledge compounds and one that redecorates.
Give it a fair comparison
The weakest evidence is before-versus-after over different periods, because time carries confounds. Stronger, in rough order:
A/B test — same period, split traffic, the only difference is the change. The gold standard, and it requires enough traffic that a real effect would show.
Before/after with a control step — did an unrelated step also move? If your whole site rose 15% that week, your fix didn't do it.
Before/after across matched periods — same weekdays, same campaigns running, enough weeks either side to absorb noise.
If you don't have the traffic for a test
Most sites don't, and pretending otherwise produces confident nonsense. Do this instead.
Prefer changes whose mechanism you understand. A validation message showing the wrong field's error is a defect. Fixing it doesn't need a control group; it needed a bug report.
Watch direction over weeks, not significance over days. A step that sits higher for six consecutive weeks is telling you something, even when no p-value will certify it.
Prefer bigger changes. If an effect is too small for you to detect, it's also too small to matter to your revenue.
Accept that you're accumulating evidence, not proof. Say so out loud. A team that knows which of its beliefs are unproven makes better decisions than one that has forgotten.
The trap of the reverted test
If a change looks negative, resist reverting immediately on a few days of data. Early numbers are noisiest, and novelty effects run both ways. Decide the observation window before you ship, and honour it.
Learning that compounds
The purpose of measuring a change isn't the change. It's that each honest result sharpens your model of your visitors, so the next fix is chosen better. Forty fixes, measured never, teach nothing. Six fixes, measured properly, teach you where your site actually leaks.
Defrixa is built around this loop: a deterministic score you can move, one fix at a time, with the effect measured at the step you changed as traffic accumulates.
Common questions
Long enough for the step to accumulate meaningful traffic and for weekday effects to average out — usually complete weeks, not days.
No. That's within the range ordinary randomness produces. Either wait, or accept it as weak evidence and say so.
Common, and invisible in the average. If you can split by device or source, do — a mobile-only regression hidden inside a positive total is a frequent and expensive outcome.