TestSuiteBench

Coding Agents Struggle to Generate Strong Test Suites from Specifications

Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller, Martin Vechev

SRI Lab ETH Zurich LogicStar.ai
Test generationCode gen.
#Modelℳ (%)𝒱 (%)Cost ($)Time (min)ℛ (%)
🥇Claude Opus 5.5 (high)56.996.53.5224.995.4
🥈Claude Opus 5 (high)55.596.110.5637.091.8
🥉Claude Opus 5.5 (medium)53.996.62.3314.994.6
4GPT 6.1 Sol (medium)51.692.90.4414.592.0
5GPT 6 Astra (medium)50.194.32.1812.492.7
6Gemini 3.8 Flash (medium)37.988.21.639.386.2
7GPT 5.6 Sol (medium)37.794.40.898.286.9
8DeepSeek V4 Flash (high)28.678.10.1419.066.8
9GPT 5.6 Luna (medium)19.374.10.034.065.4

ℳ (mutation score. main metric): % of mutants killed by the generated tests. 𝒱 (validity rate): % of generated tests that pass on the reference. ℛ (reference pass rate): % of reference tests passed by the agent's own implementation. See details.

What is TestSuiteBench?

TestSuiteBench asks coding agents to write a complete test suite for a real application from its specification alone. It contains 165 open-source applications, each with a detailed specification, a reference implementation, a reference test suite, and requirement-targeted mutants.

Task

Overview: a coding agent writes a test suite from a specification; the suite is run on the reference executable and on mutants to compute the validity and mutation scores.
Each instance has a specification, a reference test suite, and mutants that each break one requirement. The agent writes a full test suite from the specification alone. We score the suite by its validity 𝒱 and mutation score ℳ.

Each instance is a real open-source command-line application written in Rust, Go, or C/C++. The agent receives only a natural-language specification of the application's behavior and must write a complete test suite for it from scratch. The reference implementation, the reference tests, and the mutants stay hidden. Agents run in a Docker container without internet access. We measure the following metrics.

Mutation score ℳ
The fraction of mutants killed by the generated test suite; this is the main metric. Only tests that pass on the reference executable count, so a test that fails everywhere cannot inflate the score.
Validity rate 𝒱
The fraction of generated tests that pass on the reference executable. Flaky tests are considered as invalid.
Reference pass rate ℛ
For comparison, the same agent also implements the application from the same specification. ℛ is the fraction of reference tests that this implementation passes.

Findings

Benchmark Construction

Top: from a GitHub repository, the pipeline obtains the implementation I, tests T, a draft specification S', mutants M and the final specification S. Bottom: the corresponding sets of admissible implementations. Conceptual view in the space of implementations: the set admitted by the tests T is aligned with the set admitted by the specification S.
Top: we obtain a reference implementation \(I^*\), executable \(E^*\) and test suite \(T^*\) from a GitHub repository \(R\). We then write a draft specification \(S'\) and mutants \(M\) that each violate one requirement, and finally refine \(S'\) into the final specification \(S\). Bottom: in the space of implementations, we define the admissible set using \(T\) and align it with the set admitted by \(S\).
  1. Implementation \(I\) and tests \(T\). We start from the reference implementations and test suites of ProgramBench and filter out flaky and resource-heavy tests.
  2. Draft specification \(S'\). A coding agent writes a specification describing every behavior tested by \(T\).
  3. Mutants \(M\). For each specification section, a coding agent modifies \(I\) to break one requirement. We refine the mutants so that 73% are killed by exactly one test in \(T\) (up from 32%).
  4. Final specification \(S\). Agents generate implementations and tests from \(S'\). We fix every failure caused by an ambiguous or contradictory specification and repeat until fewer than 5% spec-related errors are detected. A final manual check follows, giving \(S\).

Citation

@misc{egashira2026testsuitebench,
      title={TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications},
      author={Kazuki Egashira and Niels Mündler-Sasahara and Mark Niklas Müller and Martin Vechev},
      year={2026},
      eprint={2610.13065},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2610.13065},
}