In brief
- Predefine the protocol and claim.
- Match tools, prompts, retries, and budgets.
- Publish failures and limitations alongside wins.
Define the claim before the run
A test of short Python fixes cannot support a claim about general software engineering. A synthetic retrieval task cannot establish real-world research reliability. State the population of tasks and conditions the result represents.
Match the conditions
Give candidates equivalent context, tools, time, retry policy, and stopping rules. Record model and harness versions. If one model requires a different interface, report the difference rather than hiding it.
Control leakage and tuning
Separate development and held-out tasks, freeze prompts before the final run, and disclose when tasks may be public training material. Repeatedly tuning on the test set turns it into a development set.
Report what would change the conclusion
Show per-task results, uncertainty, exclusions, failed runs, sensitivity to scoring choices, and counterexamples. Explain where the benchmark is likely not to transfer.
The strongest evaluation report makes it easy for a skeptical reader to find the boundary of the claim.
Frequently asked
Questions, answered plainly.
Are public leaderboards useful?+
Yes as one signal, but deployment decisions should include task-shaped evaluations and current operating constraints.
What is benchmark contamination?+
It is the risk that evaluation items or close variants appeared in training or tuning data, making results less representative of novel tasks.
Should failed runs be excluded?+
Only under a predefined rule that applies equally. Report exclusions and the sensitivity of results to them.
Sources and next paths