In brief
- Freeze task inputs and environments.
- Use executable gates plus human review.
- Publish per-task results and failure modes.
Write the decision the evaluation must support
Are you selecting a default model, a low-cost lane, an architecture reviewer, or a fallback? Each decision needs different tasks and thresholds. A broad benchmark cannot answer every operational question.
Sample the work, not the marketing categories
Draw tasks from actual repository work while removing secrets and unstable dependencies. Record the starting commit, environment, prompt, tools, time budget, and success checks.
- Bug diagnosis with a known cause
- Focused implementation with regression tests
- Cross-file refactor
- Code review with seeded defects
- Ambiguous request requiring scope judgment
Layer objective and subjective gates
Compile, typecheck, lint, and tests catch concrete failures. Human review catches needless churn, weak explanations, unsafe assumptions, and changes that technically pass while missing intent.
| Layer | Evidence |
|---|---|
| Execution | Commands and exit codes |
| Correctness | Targeted plus regression tests |
| Review | Blind rubric and comments |
| Efficiency | Time, tokens, attempts |
| Safety | Scope, secrets, destructive actions |
Report distributions and failure cases
Show per-task outcomes, confidence intervals when meaningful, and representative failures. Averages can hide a model that is excellent on small tasks and unreliable on cross-cutting changes.
The benchmark is valuable when another engineer can rerun it and disagree with a specific judgment.
Frequently asked
Questions, answered plainly.
Should benchmark reviewers know the model name?+
Blind review when practical. Model identity can bias perceived quality, especially for prose explanations.
Are unit tests enough to score coding models?+
No. Tests are essential but can miss scope, maintainability, security, and whether the implementation satisfies the user's actual goal.
How often should evaluations rerun?+
Rerun when model versions, tools, prompts, repositories, or deployment policy changes enough to affect the decision.
Sources and next paths