Instrument 07 · Correlated Error

How often does a clean sample hide a failing class?

It's like drawing tickets from a drum where some percentage are losing tickets — you can easily draw an all-winners run and wrongly conclude there are no losing tickets in the drum at all. Adjust the sample size and the true failure rate below, then re-run the simulation to see how often a "clean" check happens purely by chance.

44%

of 1,000 simulated 20-item checks came back "all clean," even though 4% of items truly had the problem.

Seed: 42 · n=20 · failing class: 4%

Example: if 4% of a supplier's parts have a hidden defect, and you inspect 20 parts, the number above is how often that check comes back "all clean" purely by chance — even though the defect rate in the full batch never changed. A clean sample this size is not proof the problem doesn't exist; shrink the sample size or the failure rate above and watch the "clean" percentage climb.

WHY THIS INSTRUMENT EXISTS

A single accuracy number can describe a tool that performs well on average while reliably missing one class of failure — the rare, consequential kind that a golden set weighted toward easy cases will not reveal.

Simulating repeated sampling against a known failing class shows how often a clean-looking evaluation run can occur even when a real weakness is present, which is the honest argument for testing composition, not just testing volume.

§2.1 AI fundamentals for quality professionals§8.1 Software testing is not AI evaluation§2.2 Why fluent is not the same as correct§8.3 Coverage, missed findings and false findings§10.4 Anomaly detection in production and test data