INSTRUMENT 09 · GOLDEN SET BUILDER

What goes in the reference set, and is it honest?

A "golden set" is the fixed batch of test cases you use to check whether an AI system is working correctly — like an answer key a teacher uses to grade a test, or the control samples a lab runs alongside real ones to prove the instrument is reading correctly. Enter how many cases of each kind you have. The mix matters: if every case in your set actually has a problem to find, the system can score well by simply always claiming to find one — like a smoke detector that just beeps constantly and is "right" every time there happens to be smoke.

Example: a weld-inspection AI. "Normal cases" are good welds it should pass. "Failure conditions" are cracked or porous welds it should catch. "No-finding cases" are welds with a cosmetic quirk (discoloration, a rough edge) that look suspicious but aren't actually defects — these prove the system can say "this is fine" instead of flagging everything it's unsure about. "Near-misses" are welds just barely within tolerance. Without enough no-finding and near-miss cases in the test set, an overzealous system that flags every weld would look accurate, because most test cases would happen to have a real problem.