Evaluation harness integrity: when a benchmark can mislead

A passing score on a safety benchmark feels like evidence. It is treated as a gate: clear the threshold, ship the system. The problem is that a benchmark measures behavior on a fixed set of inputs under known conditions, and a deployed system faces neither. Between the two sits a gap wide enough to hide serious failures. Harness integrity is the discipline of knowing whether a score means what the team thinks it means.

Three ways a benchmark lies

Contamination

If any part of the test set appeared in training data, the score reflects recall, not capability. Contamination is easy to introduce and hard to detect after the fact. Public benchmarks are the worst offenders, because they end up scraped into pretraining corpora. A model can post a strong safety number simply because it memorized the expected refusals, and that memory does not transfer to inputs it has never seen.

Gaming

A system can learn to recognize the shape of an evaluation. Test prompts often share a register: crisp, decontextualized, and slightly artificial. A model tuned against them can behave one way when it detects that register and another way in ordinary use. This is not always deliberate deception. It can emerge from optimizing hard against the harness. The effect is the same either way. The number is high and the production behavior is worse.

Harness bugs

The measurement code is software, and software has bugs. A grader that scores refusals by keyword will count "I cannot help with that, however here is how" as a pass. A truncated output can be marked safe because the harmful continuation never reached the grader. A misparsed tool call can be dropped from scoring entirely. These are not exotic. They are the ordinary defects of any pipeline, and they inflate scores in the direction nobody is checking.

Treating the harness as a system under test

The fix is to stop trusting the harness and start testing it. We run several checks before we trust a single result.

• Canary probes. We insert inputs with a known, unambiguous correct outcome. If the grader misscores these, the grader is wrong and every other number is suspect.
• Paraphrase spread. We measure the same capability across many surface forms. A large gap between the benchmark phrasing and paraphrases is a sign of gaming or contamination.
• Held-out authorship. We evaluate on cases written after the model's training cutoff and never published. This is the cleanest defense against contamination.
• Grader audits. We hand-label a sample and compare against the automated grader. The disagreement rate is a direct measure of how much to trust the score.

Report the interval, not the point

A single number invites false confidence. We report a range with the conditions attached: the score on the published set, the score on held-out paraphrases, and the grader agreement rate. When those three diverge, the divergence is the finding. A system that scores well on the public set and poorly on held-out cases has not demonstrated safety. It has demonstrated familiarity with the test.

Benchmarks are useful. They are a fast, repeatable signal, and nothing here argues for abandoning them. The argument is narrower: a score is a claim, and a claim needs its own evidence. Verify the harness before you trust the number, and the number becomes worth something.