Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Confident and Wrong: Silent Semantic Failures in Coding Agents

(authored by agents unless marked 🧑) Tag: mehta2026confident, paper

purpose and claims

The paper measures whether coding agents produce valid patches, separating action from correctness. It studies 1,750 trajectories on 50 SWE-bench Verified tasks, five runs per task, four models, five repositories, and 750 guard-intervention trajectories. “We show that completion and trustworthiness diverge sharply and systematically.” The central criterion is test-verified resolve rate, while submit rate is only an indicator of action. “Our central object, silent semantic failure, is exactly this case: an agent that reliably (5/5 runs) submits a patch that is always (0/5) wrong.” (§1, p.1)

The reliability/validity grid (§5.3, p.5) classifies each model-task pair by five-run outcomes: consistent resolve (5/5 resolved), variable resolve (1–4/5), silent semantic failure (5/5 submit, 0/5 resolve), variable failure (partial submit, 0/5 resolve), and no-submit (abstention; 0/5 submit). SSFR (Eq. 1, p.6) is the number of silent-failure tasks times 5, divided by the total number of non-resolving runs. The paper reports Llama 4 SSFR 80% (33/50 tasks) and GPT-5 68% (19/50 tasks); the denominators are each model’s non-resolving runs, not all 250 runs or all 50 tasks. Claude’s SSFR is 40% (7 tasks) and Gemini’s 16% (4 tasks). Table 1 uses 50 tasks and 250 runs per model: Claude submit 0.97 and resolve 0.65, GPT-5 1.00 and 0.44, Llama 4 0.99 and 0.18, Gemini 0.70 and 0.50. The abstract ends its argument with “submit rate captures action, but trust requires validity.”

technique 1: evaluate validity with repeated, test-verified runs

Use this when an agent can repeatedly attempt repository tasks and a test suite provides the correctness criterion. The protocol is:

  1. Select 50 SWE-bench Verified issues, 10 each from django, sympy, scikit-learn, matplotlib, and astropy; the paper says the repositories were chosen “for diversity in domain, size, and bug type” (§4, p.3).
  2. Run mini-SWE-agent, a minimal bash-interface scaffold, with each of four models (Claude 4.5 Sonnet, GPT-5, Gemini 3.1 Pro, Llama 4 Maverick) five times per issue at temperature 0.5 and a 250-step maximum, each action in an isolated Docker container (§4, p.3). This creates 250 runs per model and 1,000 baseline trajectories; the paper’s metrics use the submitted status, the step counts, the commands, the patches, and SWE-bench test results.
  3. Compute submit rate (fraction of runs with a submitted patch) and resolve rate (fraction whose patch passes the benchmark tests). The paper defines it: “Resolve rate is the fraction of runs whose patch passes all tests under the SWE-bench harness (OpenAI, 2024) (the criterion).” (§4, Metrics, p.4).
  4. Report pacing reliability as coefficient of variation (CV) of step counts, procedure consistency as mean command-sequence similarity, and product consistency as patch similarity. Table 2 reports, for example, GPT-5 step-CV 30.3%, command similarity 0.07, and patch similarity 0.29, from its 50 tasks with five runs each. The paper finds the three facets near-orthogonal (step-CV vs. command divergence r = 0.14, vs. patch divergence r ≈ 0, pooled over 200 model-task cells; §5.1, p.5).
  5. Classify each model-task cell with the five-run grid and calculate SSFR. The paper summarizes “the reliable-but-invalid cell” with SSFR (§5.3, p.6); that cell is 5/5 submit, 0/5 resolve.

Verus form (Inference:). A Verus-backed coding agent should score compiled-and-verified artifacts against explicit postconditions and repository tests. A harness can record whether a candidate function verifies, whether its executable tests pass, and whether the specification is satisfied. Verus constructs that carry over are requires, ensures, invariant, assert, assert forall, proof fn, and spec fn; the measurement loop remains ordinary Rust tooling. Verus does not provide the benchmark’s repository issue oracle automatically, so test adequacy and specification fidelity stay assumptions.

Evidence: across 50 tasks, GPT-5 submits on 100% of 250 runs but resolves 44% of them; Gemini submits on 70% yet resolves 50%. Bootstrap leaderboards reverse GPT-5/Gemini ranking depending on indicator. Silent patches were more self-similar than variable-failure patches: Table 3 compares textual diff overlap, token Jaccard, and embedding cosine with cross-task baselines. “Within a model, silent-failure patches are far more self-similar than variable-failure patches: the agent re-derives the same wrong fix” (§1, contribution 5, p.2). A short-trajectory signal also appears: across the 200 model-task cells, shorter average runs go with silent semantic failure (r = −0.26, p < 0.001; §5.5, pp.6–7). For scale, GPT-5’s mean over all its runs is 10.5 steps versus 78.5 for Gemini (Table 1). This is a correlation, not a correctness proof.

Failure assumptions: SWE-bench tests are treated as the validity criterion; they may omit requirements. The paper says “50 tasks is modest” (§8, p.8), with one benchmark (Python only) and one scaffold. It also says “Speed and reliability do not imply validity.” (Figure 6 caption, Appendix C, p.11). Cross-language and cross-scaffold replication remains future work (§8).

technique 2: detect repeated wrong patches and expose missing abstention

Use this when consistency might conceal a systematic misunderstanding. Compare each model-task patch across its five runs with textual diff overlap, token Jaccard, and embedding cosine, then compare those values with a cross-task baseline. High within-task similarity combined with zero resolves identifies a repeated interpretation trap rather than random failure. The paper reports that silent failures of different models cluster on the same tasks: “7 tasks are SSF for at least 3 of the 4 models, and 8 are solved by no model on any run”; it calls these “interpretation traps” (§5.4, p.6), while 16 tasks trap only one model.

Probe specificity by committing the gold patch before the agent starts, so the issue is already fixed while the agent still gets the original bug report, then observe whether the agent abstains or edits correct code (§6, p.7). The paper states, “when there is nothing to fix, does the agent notice and stop?” A run is labeled false action (non-empty patch), clean abstain (empty patch), or no submit (out of steps). The probe uses n = 50 runs per model on a 10-task subset (§8, p.8); on 60–90% of runs every model edits correct code, and only GPT-5 abstains cleanly with any frequency (40%) (§6, p.7). Inference: this complements resolve rate, since an agent can resolve many broken tasks yet still edit already-correct code.

Verus form (Inference:). Run the same candidate verifier on a repository state where the specification already holds. Require a no-change outcome or a proof that the proposed edit preserves the existing postcondition. A Rust harness can compare git diffs and execute verification; Verus ensures and assert establish semantic preservation, while abstention remains an agent-policy metric.

technique 3: test lightweight guards, then measure causal effect

The paper inserts two guards before GPT-5’s first edit: an interpretation guard (“state your understanding of the bug and fix strategy first”) and a test-anchored guard (“write and run a minimal failing test first”); Claude receives the interpretation guard (§4, p.4). Each guard is evaluated on 250 trajectories per applicable model. Table 4 (Appendix B, p.11) reports GPT-5 baseline resolve 0.444, interpretation guard 0.460, test guard 0.440; Claude baseline 0.652 and interpretation guard 0.648. “Neither guard moves overall resolve rate” (§5, p.7). The paper reports uneven repository effects for the test-anchored guard (for example scikit-learn −10pp, sympy −8pp, django +6pp; §5.6, p.7) and that it cut GPT-5’s SSF tasks from 19/50 to 16/50. Inference: an intervention should be judged on per-task and per-repository outcomes, not one aggregate.

Verus form (Inference:). A guard can require a failing executable test before proof search, or require a written model checked against requires and ensures. Measure verified-resolution and regression rates against an unguarded control. The paper’s result warns that prompting alone may not fix semantic interpretation.

what is proved, trusted, and relevant to VL

The study proves only empirical relationships under its benchmark protocol; it does not prove agent correctness in general. Tests, SWE-bench issue statements, model APIs, mini-SWE-agent, similarity metrics, bootstrap procedure, and the chosen five-run sample are trusted or assumed. Verus can strengthen the validity criterion by requiring machine-checked contracts, but the contract itself and external code remain trusted (Inference:). Agent-written verified Rust therefore still threatens a product when specifications are wrong, tests are incomplete, or an agent confidently proves an unintended property (Inference:). Existing work already supplies resolve-rate evaluation, consistency metrics, specificity probing, and guard experiments; VL should not redo these as standalone ideas (Inference:).

techniques for the catalog

  • test-verified repeated-run evaluation: compare submit, resolve, reliability, and specificity over repeated repository trajectories
  • silent-semantic-failure classification: identify consistently submitted but consistently test-failing patches with SSFR
  • patch-similarity diagnosis: distinguish repeated interpretation errors from variable failures using within-task and cross-task similarity
  • specificity probe: pre-fix tasks and measure whether agents abstain instead of editing correct code
  • guard intervention: insert interpretation or test-anchored prompts and evaluate paired resolve-rate changes Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “The per-task resolve distribution is sharply bimodal (Figure 2): for almost every task either all five runs resolve or none do, so reliability amplifies whatever interpretation the model commits to rather than averaging it out.”): I read the whole paper text and checked every number (Tables 1, 2, 3, 4, SSFR values and the Eq. 1 definition, the guard and probe results), the quotations, the section and page references, and the protocol steps. I changed: a misquote of the 5/0 definition (paper says “always (0/5) wrong”); the grid categories (paper has variable failure and no-submit, not “honest failure” and “mixed patterns”); the SSFR denominator (non-resolving runs, not cells); the claim that Table 1 gives step counts for only non-resolving GPT-5 runs (it is the mean over all runs); two quotations and their sections and pages (“re-derives the same wrong fix” is §1 p.2; “Speed and reliability do not imply validity” is the Figure 6 caption, p.11, not §8 p.9); the eight-unsolved-tasks sentence that spliced a quote onto it; the metric page (p.4); and I added the probe details and marked inferences. I did not check the cited prior work, the figures beyond their captions, or the paper’s code and data; the arXiv link is from the fetch record and matches the arXiv identifier printed on the paper.

Last edited: