The problem
Most agent metrics measure activity: tokens burned, tasks marked complete, sessions run. An agent can score well on all of them while producing wrong, unsafe, or worthless output, and a broken evaluator looks identical to a passing one. If the grader rewards the mechanism, the mechanism is what you get.
The approach
I defined a north-star objective for my agent harness and decomposed it into eight measurable goals, each stated as an outcome a user would notice. Each goal is graded separately, with rules designed against Goodhart's law, the observation that a measure stops being useful once it becomes a target.
- No composite score. A run reports the worst status across all goals. One failing goal cannot hide behind seven passing ones, and there is no single number to optimize toward.
- Counter-metrics. Each goal carries a check for the obvious way to game it, so improvement on the target number is weighed against the damage it might be doing elsewhere.
- Fail closed. An extractor crash, a parse failure, or missing data fails the run. A silent evaluator failure is the most dangerous outcome in the system, so absence of evidence is treated as failure, not success.
What gets measured
The harness works from real material: session transcripts, secret scans, service health checks, and baselines that catch regressions between runs. The current quick evaluation exits clean with a worst status of "warn", and it is reported as exactly that, a warning, not a pass.
Adversarial audits
The harness itself gets audited. A standing review asks where the evaluation rewards hand-built structure over actual model capability, and where a metric has quietly become a target. Findings from those audits change the harness.
Limits
Some behavioral dimensions remain unmeasured, and the goal list says so rather than padding itself. This is an engineering control for my own agent estate, not a safety certification, and I do not present it as one.