Most enterprise AI projects build evaluation last. That is the wrong order. The harness is the cheapest artifact to build and the most expensive to add later.
The shape of a real harness
Four artifacts your team owns:
- Golden set - 20 to 200 hand-curated, hand-labeled examples per use case.
- Rubric - pass/fail or numeric, written down, versioned in the repo next to the runner.
- Runner - code, in CI, that scores every output against the rubric.
- Cost gate - a parallel rubric that scores cost per request and fails the build if it regresses past a threshold.
When to start
Today. The act of writing the golden set is how your team agrees on what the AI is for. Postponing it means the team disagrees, silently, until production forces the disagreement into the open.
Anti-patterns
- "We will use the LLM to grade itself." Sometimes. Not for safety-critical rubrics. Have humans grade the first hundred outputs by hand. That is your reference.
- "We will use a public benchmark." Benchmarks measure the model. Your harness measures your system on your use case. Different artifact, different job.
- "We will add evals after launch when we have time." You will not have time.
The minimum viable eval
If you have nothing today, ship this in one week:
- Pick three use cases.
- Write five passing, three failing, two ambiguous examples per use case.
- Write a one-line expected behavior next to each.
- Build a runner that emits a pass/fail table on stdout.
- Wire it into CI as a non-blocking check.
By week two, make it blocking.
The lesson
Evals are not overhead. Evals are the only thing that lets your team ship AI changes safely. The teams that build them first ship faster. The teams that postpone them ship into incidents.