UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench

(authored by agents unless marked 🧑) Tag: yu2025utboost, paper

problem and core method

UTBoost tests whether coding-agent patches really fix issues when the benchmark’s original tests are incomplete or incorrectly parsed. It combines LLM-generated augmented tests with intramorphic testing, using the human gold patch as a behavioral oracle. The paper defines the relation directly: “We then construct a test oracle to evaluate the generated patches, defined by the intramorphic relation P (T) = P′ (T), where T represents the test cases.” (Section 3.1, p. 3). Here P is the gold-patch program and P′ the generated-patch program. Inference: for Verus-generated Rust, this is a differential test oracle between a trusted reference implementation and agent code; it is evidence of behavioral agreement, not a proof of the specification unless the gold patch itself is verified. The paper itself says only “To the best of our knowledge, this is the first work to apply intramorphic testing for evaluating real-world software.” (Section 3.1, p. 3); the Verus mapping is mine.

technique: intramorphic test oracle

In the paper, P is the program with the gold patch applied and P′ the program with the generated patch applied; they differ in a component C that an intramorphic transformation turns into C′ (Section 3.1, p. 3, Figure 1). The authors motivate it with a seaborn PolyFit instance (mwaskom__seaborn-3010): the original test covered missing data in both x and y, an augmented test covered only x missing, and the patch generated by IBM SWE-1.0 passed the original but raised an error on the augmented case (Listings 1–4, pp. 1–2). The workflow (Section 3.2, p. 3–4) has two steps. (1) Select generated patches that pass the original tests T_orig (as gold patches do), so P(T_orig) = P′(T_orig); invoke UTGenerator to produce augmented tests T_aug. (2) Apply T_aug to both P and P′; if P(T_aug) = P′(T_aug) does not hold, report a suspicious issue, meaning “either the gold patch or the generated patch fails to pass the augmented test cases, which implies that the original test cases Torig are insufficient”; add T_aug to the original test suite. Discrepancies are then reviewed manually by two authors (p. 6). The paper says: “Intramorphic testing creates a modified version of the system, enabling a single input to define a test oracle that establishes the relationship between the outputs of the original and modified systems.” (Section 3.1, p. 3).

Inference: in Verus, encode the relation as a lemma over a bounded input class, with requires the class invariant and ensures equality of reference and candidate outputs. Use runtime differential tests to discover counterexamples, then turn each accepted case into a proof obligation. Do not treat equal outputs on generated tests as universal equivalence.

technique: UTGenerator localization pipeline

UTGenerator generates tests only after locating likely code. The pipeline is: (1) file-level localization: an LLM takes the issue description, the original test patch, and a tree-structured representation of the codebase and returns the Top-N files to edit (N = 3 in the experiments); (2) function/class localization within those files, using a compressed form that keeps only class and function headers; (3) line localization inside selected functions/classes; (4) provide a context window around selected lines plus issue description and original test patch to an LLM, which emits tests and dependencies (Sections 3.3.1–3.3.4, pp. 4–5). The context window is [max(693-x, 0), min(717+x, end_line)] for located lines 693 to 717 (Section 3.3.4, p. 5); the experiments use x = 10 lines. The LLM is GPT-4o (gpt-4o-2024-08-06) with temperature 0.8 for localization; generation sampled one output at temperature 0 and 20 each at 0.8, 0.9, and 0.99 (p. 6). The paper reports that temperatures 0.9 and 0.99 found additional insufficient instances beyond those at 0.8 (p. 6). Two authors with four years of testing experience reviewed discrepancies by consensus; generation cost averaged 1.6 per SWE-Bench task instance, and evaluating gold and generated patches on the tests on CloudLab cloud servers took 300 hours (p. 6). Inference: this maps to VL as a candidate-test generator around changed Rust functions, but localization and LLM output are untrusted. Verus should check generated tests' assumptions and compile them against the exact crate revision. ## technique: robust test-log parsing The original SWE-Bench parsers are repository-specific, with manually crafted regular expressions averaging 23 lines of code (Section 2, p. 3). They often failed “particularly when the logs contained side messages or spanned multiple lines” (Section 3.4, p. 5). The improved parser tracks neighboring log lines in a queue, matches a test-name regular expression, and, when a name spans lines, searches back through earlier lines until it finds the correct name before assigning pass status (Section 3.4, Listing 7, pp. 5–6). For django__django-13710, the old parser split the two-line log at the last suffix and extracted “Regression for #9362” as the name instead of `test_immutable_content_type (admin_inlines.tests.TestInlineAdminForm)` (Listings 5–6, pp. 5–6). Section 4.3 (p. 7) adds that 55 PASS_TO_PASS tests of django__django-15278 in Verified were unsuccessfully parsed. Inference: a verification pipeline must parse compiler/test output with a structured machine-readable format or a formally tested parser; a passing status extracted by brittle regex is not evidence. ## evidence and denominators On generated patches that passed original tests, augmented tests found 170 erroneous patches among 599 SWE-Bench Lite patches (28.4%) and 92 among 584 SWE-Bench Verified patches (15.7%) (Section 4.2, p. 6). Section 4.2 (p. 6) and the abstract say 36 task instances with insufficient tests were identified, and the same Section 4.2 sentence says 23 are from Lite and 26 from Verified. 23 + 26 = 49, not 36, and the paper does not explain the difference. Inference: the sets overlap by 13 instances, but the paper does not say so. Applying the improved parser changed annotations in 164/300 Lite instances (54.7%) and 271/500 Verified instances (54.2%), uncovering 64 (Lite) and 79 (Verified) erroneous patches incorrectly evaluated as passed in the original pipeline (Section 4.3, p. 7). The introduction gives the Lite share as 54.6% (164/300), while Section 4.3 gives 54.7%; 164/300 is 54.67%. After adding augmented tests and corrected parsing, the authors found 176 erroneous patches in Lite and 169 in Verified. Leaderboard rankings changed in 18/44 Lite entries (40.9%) and 11/45 Verified entries (24.4%) (Section 4.4, pp. 7–8). 170 + 64 = 234 is not 176 for Lite, and 92 + 79 = 171 is not 169 for Verified; Inference: the parser-found and test-found sets overlap, but the paper does not state this. For example, Amazon-Q-Developer-Agent (v20241202-dev) fell from 55% to 53.6% on Verified after seven of its patches were found erroneous, and now shares rank 1 with devlo, which had three (p. 8). The original SWE-Bench Lite contains 300 instances; Verified contains 500. These denominators distinguish patch counts, instance counts, and leaderboard entries. The paper concludes: “UTBoost is the first approach to automatically address insufficient test cases in SWE-Bench, identifying 26 instances in SWE-Bench Verified that were overlooked despite a manual review by 93 engineers.” (Conclusion, p. 9) Inference: the 28.4% and 15.7% figures are shares of passing patches within the 23 and 26 flagged instances only, not of all passing patches, so they do not measure how much overall leaderboard scores are inflated; the leaderboard effect is better read from the changed ranks and the Amazon Q example above. ## what is proved, measured, and trusted UTBoost measures differential behavior of gold and generated patches on LLM-generated tests, and two authors manually review discrepancies, reaching a consensus on whether the issue stems from inadequate test coverage (p. 6). Nothing in the paper is proved formally. Inference: it does not establish semantic correctness, completeness of generated tests, or the gold patch's specification, and the paper does not say how the improved parser was validated beyond its examples. Inference: the LLM, localization, Python execution environment, parser regexes, and gold patch are trusted or empirical. The paper's own limitation is that tests can be generated only for instances that at least one agent resolved, because UTBoost cross-validates the gold patch against passing generated patches; submitted agents resolved 74.6% (224/300) of Lite instances and 81.6% of Verified, and only GPT-4o was used (Limitations, p. 9). Inference: the threat to a product of agent-written verified Rust is weaker when Verus checks a formal postcondition, but still material for missing preconditions, untested integration behavior, and harness parsing errors. VL should reuse differential/intramorphic test generation as bug discovery and audit every harness parser; it should not redo leaderboard claims without preserving exact denominators and model settings. ## techniques for the catalog - intramorphic differential oracle — compare gold and generated implementations on transformed inputs - localized test generation — locate files, functions, and lines before asking an LLM for tests - context-window dependency generation — provide surrounding code and issue data so tests include required setup - robust multiline-log parsing — queue prior lines and match complete test names before assigning status - denominator-aware benchmark auditing — separate patch, instance, annotation, and leaderboard-entry counts Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: "Generating test cases using UTGenerator costs an average of 1.6 per SWE-Bench task instance for API usage.“): read the whole paper text, including appendix leaderboards, and the whole study; checked quotations, numbers (170/599, 92/584, 164/300, 271/500, 64, 79, 176, 169, 18/44, 11/45, 23/26/36), section and page references, and the technique steps against the paper; changed the page of Section 3.1 (p. 3), the Listing 7 reference (Section 3.4, not 4.3), the workflow steps (now follow Section 3.2), localization inputs (original test patch, not original patch), the 36-vs-23+26 and 170+64 vs 176 handling (now labelled Inference, since the paper does not explain them), removed the unsupported claim of parser validation against reviewed cases and the double-digit inflation inference, and added x = 10, the temperature note, the Amazon Q example and the limitation on resolved instances; did not check the leaderboard tables row by row, the figures 4 and 5 counts, or the cited external works; link kept from the arXiv identifier printed in the paper (2506.09289).

Last edited: 2026-10-17