Guarding against regressions

The scariest result isn’t the failure you expected; it’s something that used to work breaking quietly. Simulation layers several mechanisms to catch regressions.

​

Experiment option: Hold-out regression

Turn on Hold-out regression in a new experiment’s options. It guards previously-correct cases against regression, at one of three levels: Off, Light, or Full (see New experiment). With it on, the experiment results show Safe on holdout N/N: the number of protected cases.

​

Reading regressions in the results

• Broke in the results summary counts previously-working cases that broke.
• Case-by-case in a candidate’s detail marks each case as improved or regressed.
• A candidate with no regressions carries the “nothing broke” badge.

The board’s Regression column collects experiments with regressions left or no confirmed improvement. Landing there doesn’t mean the experiment is discarded; it means re-run with different options, or verify the candidates further.

​

The deep test’s Holdout check

The last layer before publishing. Run the Holdout check in a candidate’s deep test to re-confirm the candidate doesn’t break previously-working cases. When the improvement holds with no regressions, the “safe uplift” badge appears (see Verify and publish).

​

The Revalidation tab in Signals

The reverse direction after shipping belongs to the Revalidation tab on the Signals page: recoveries to reclaim (bad→good) and holdout regressions (good→bad) (see Revalidation).

​

Learn more