Eval Harness: Capability, Regression and Repeated Reliability

What it does

ECC’s eval harness treats evaluation as a development contract. Capability cases ask whether a new behavior works; regression cases protect established behavior. Code graders check deterministic properties, model graders assess rubric-based judgments and human review covers ambiguous or consequential outputs. The source distinguishes pass@k, where at least one trial succeeds, from pass^k, where every trial succeeds. That distinction matters for unattended agents: a workflow that eventually gets lucky is different from one that behaves consistently. Versioned definitions and reports make comparisons interpretable across changes.

When to use it

• Compare two prompt versions for structured invoice extraction using synthetic documents.
• Protect refusal and permission boundaries while adding a new agent tool.
• Measure whether an automated code task succeeds consistently across repeated attempts.

Step-by-step workflow

1. Define the task, permitted tools, success rubric and representative capability and regression cases before implementation.
2. Choose deterministic graders where possible and specify a review rubric for judgments that require a model or person.
3. Freeze inputs, model settings and tool conditions; record trial counts, budgets and how invalid runs are handled.
4. Run baseline and candidate with the same protocol, retaining outcomes and failure categories rather than only aggregate scores.
5. Review regressions and uncertainty, then report promotion criteria and any cases that require human acceptance.

Example request

Design an eval for our invoice-extraction prompt using 30 synthetic invoices with known totals and currencies. Include missing fields, ambiguous dates and instructions embedded inside invoice text. Grade schema and arithmetic deterministically; send only genuinely ambiguous interpretation cases to a documented human rubric. Compare baseline and candidate over repeated trials with fixed budgets. Report first-attempt success and all-trials success separately; do not count a retry as first-pass reliability. Keep a held-out set and list examples the grader cannot judge safely.

Worked example

A candidate extracts the right total on one of five attempts but invents a currency on the other four. A metric that records only whether any attempt passed can make the candidate look useful for supervised exploration. A scheduled accounting workflow needs a different conclusion because it cannot silently try until it gets a favorable answer. The report therefore shows individual trial outcomes, deterministic field checks and all-trials success alongside the exploratory measure. Failures involving instructions inside the document remain visible as a separate security-relevant category instead of disappearing inside an average.

What you get

Versioned eval definitions, grader rules, per-case outcomes and a baseline comparison with explicit reliability and coverage limits.

How to evaluate the result

• Capability and regression results remain separate and traceable to cases.
• Retry-based success is not presented as consistent first-attempt reliability.
• Held-out inputs and grader uncertainty are documented.

What you need

Representative permitted data, stable expected answers or a clear human rubric, and controlled access to the agent under evaluation. Evaluation calls may consume provider quota.

Setup and activation

The reviewed ECC source provides an .agents skill layout. Consult the repository README for the chosen client, retain referenced supporting files and explicitly activate only the workflow needed. This catalogue read source files; it installed and executed nothing. Client compatibility depends on available tools and instruction loading, and was not runtime-tested here. Before installation, scan the exact source version, review permissions and use disposable non-sensitive examples.

Limits and considerations

A model grader can share biases with the model being tested. Small samples and reused training examples exaggerate confidence. Source metric notation needs a stated trial protocol; do not infer real-world failure probability from a handful of repeated runs or upload private datasets without permission.

Related workflows

• Use E2E Testing for concrete browser interactions.
• Use Verification Loop to connect eval results with deployment acceptance.

License and reuse

MIT in the selected SKILL.md and repository LICENSE. Preserve notices; external packages, services and example dependencies retain their own terms.

Further reading

• Reviewed skill at immutable commit
• Reviewed publisher README
• Reviewed MIT license