| Test generation | Code gen. | |||||
|---|---|---|---|---|---|---|
| # | Model | ℳ (%) | 𝒱 (%) | Cost ($) | Time (min) | ℛ (%) |
| 🥇 | 56.9 | 96.5 | 3.52 | 24.9 | 95.4 | |
| 🥈 | 55.5 | 96.1 | 10.56 | 37.0 | 91.8 | |
| 🥉 | 53.9 | 96.6 | 2.33 | 14.9 | 94.6 | |
| 4 | 51.6 | 92.9 | 0.44 | 14.5 | 92.0 | |
| 5 | 50.1 | 94.3 | 2.18 | 12.4 | 92.7 | |
| 6 | 37.9 | 88.2 | 1.63 | 9.3 | 86.2 | |
| 7 | 37.7 | 94.4 | 0.89 | 8.2 | 86.9 | |
| 8 | 28.6 | 78.1 | 0.14 | 19.0 | 66.8 | |
| 9 | 19.3 | 74.1 | 0.03 | 4.0 | 65.4 | |
ℳ (mutation score. main metric): % of mutants killed by the generated tests. 𝒱 (validity rate): % of generated tests that pass on the reference. ℛ (reference pass rate): % of reference tests passed by the agent's own implementation. See details.
What is TestSuiteBench?
TestSuiteBench asks coding agents to write a complete test suite for a real application from its specification alone. It contains 165 open-source applications, each with a detailed specification, a reference implementation, a reference test suite, and requirement-targeted mutants.
Task
- 165applications
- 5.6kwords per specification
- 679reference tests per app
- 74 average mutants per app
Each instance is a real open-source command-line application written in Rust, Go, or C/C++. The agent receives only a natural-language specification of the application's behavior and must write a complete test suite for it from scratch. The reference implementation, the reference tests, and the mutants stay hidden. Agents run in a Docker container without internet access. We measure the following metrics.
- Mutation score ℳ
- The fraction of mutants killed by the generated test suite; this is the main metric. Only tests that pass on the reference executable count, so a test that fails everywhere cannot inflate the score.
- Validity rate 𝒱
- The fraction of generated tests that pass on the reference executable. Flaky tests are considered as invalid.
- Reference pass rate ℛ
- For comparison, the same agent also implements the application from the same specification. ℛ is the fraction of reference tests that this implementation passes.
Findings
- Agents write weak test suites. The best agent kills only 56% of mutants, and four of six agents stay below 40%.
- The tests are valid, just not strong. Over 90% of generated tests pass on the reference implementation for frontier models, and implementations the same agents build from the same specifications pass up to 93% of the reference tests.
- Model-generated test suites are small. Generated suites are smaller than the reference suites and miss the input combinations needed to expose broken requirements.
Benchmark Construction
- Implementation \(I\) and tests \(T\). We start from the reference implementations and test suites of ProgramBench and filter out flaky and resource-heavy tests.
- Draft specification \(S'\). A coding agent writes a specification describing every behavior tested by \(T\).
- Mutants \(M\). For each specification section, a coding agent modifies \(I\) to break one requirement. We refine the mutants so that 73% are killed by exactly one test in \(T\) (up from 32%).
- Final specification \(S\). Agents generate implementations and tests from \(S'\). We fix every failure caused by an ambiguous or contradictory specification and repeat until fewer than 5% spec-related errors are detected. A final manual check follows, giving \(S\).
Citation
@misc{egashira2026testsuitebench,
title={TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications},
author={Kazuki Egashira and Niels Mündler-Sasahara and Mark Niklas Müller and Martin Vechev},
year={2026},
eprint={2610.13065},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2610.13065},
}