In brief
- Evaluate on your repositories and task mix rather than a single public score.
- Separate correctness, review burden, latency, and cost.
- Blind judging reduces brand and presentation bias.
There is no context-free winner
Coding performance changes with language, repository size, tool access, test quality, and task ambiguity. A model that excels at isolated functions may struggle to navigate a large codebase; a powerful reasoner may be unnecessary for mechanical edits.
Choose the evaluation before you choose the model. Otherwise the test quietly changes to favor the answer you already wanted.
Build a matched task set
Use recent, representative work that has an auditable outcome. Include small fixes, cross-file changes, debugging, test writing, and one ambiguous task that requires asking the right question.
- Keep prompts and tool permissions identical
- Start from the same commit
- Use the same time and retry budget
- Hide model identity from reviewers
Use a scorecard that reflects engineering cost
Pass rate alone hides review effort and unsafe changes. Track the whole path from prompt to mergeable patch.
| Dimension | Measure | Weight idea |
|---|---|---|
| Correctness | Targeted and regression tests | Highest |
| Patch quality | Review comments and unnecessary churn | High |
| Autonomy | Interventions required | Medium |
| Latency | Time to accepted patch | Medium |
| Cost | Total tokens and retries | Context-dependent |
Route by task after the baseline
If the results separate by task type, routing can outperform a single default. Use a fast model for narrow edits and reserve a deeper model for architecture or difficult debugging. Keep a simple default until measured gains justify routing complexity.
Frequently asked
Questions, answered plainly.
Which model family is best for coding?+
There is no universal answer. The best choice depends on your repositories, tasks, tools, quality bar, and latency budget.
How many tasks should a coding evaluation include?+
Start with enough representative tasks to expose different failure modes, then expand as decisions become higher stakes. Repeatability and matched conditions matter more than a headline count.
Should cost be part of the quality score?+
Track it separately first. Combining cost and quality too early can hide whether a model is cheaper because it is efficient or because it is failing.
Sources and next paths