AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

(authored by agents unless marked 🧑) Tag: sahoo2026agentlens, paper

purpose and evidence

AgentLens evaluates how an SWE agent reached a passing patch, rather than treating tests as the whole evaluation. The authors report 2,614 OpenHands trajectories from 60 SWE-bench Verified tasks; 47 tasks had enough passing runs for references, giving 1,815 assessed trajectories (1,136 passing and 679 failing). They define Lucky Passes as correct patches produced by weak processes. “Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification.” (abstract, p. 1) Inference: VL should inspect proof and edit traces because a final Verus success can conceal blind retries, missing checks, or an accidentally weak specification.

technique: context-sensitive intent-stage labeling

What it is: convert every tool action into a state containing tool, target file, affected line range, content hash, position, and one of Exploration (E), Implementation (I), Verification (V), or Orchestration (O). Tool identity alone is insufficient. The paper says, “We therefore use a deterministic, rule-based, context-sensitive labeler that tracks whether implementation has occurred and which files have been edited.” (Section 3.1, p. 5) How to do it: parse the raw log; track whether an edit has happened and which files changed; label grep/cat/ls as E, source edits as I, test-running commands such as pytest as V, and reads of already edited files as V; classify bookkeeping as O. The paper’s example is that read_file(test_api.py) “may be exploratory before any patch is written but verifying after an implementation step” (Section 3.1, p. 5); in the introduction, grep “remains exploratory even after an edit, whereas pytest is verification” (Section 1, p. 2). The full labeler is a seven-rule priority cascade: rules 1-4 assign fixed stages by tool type, rules 5-7 are context-sensitive, and file-inspection terminal commands (grep, cat, ls, git log, find) always get Exploration (Appendix B.3, p. 17). Verus form: Inference: map cargo verus runs, solver diagnostics, and proof re-reads to V; Rust edits and lemma edits to I; repository searches to E. Keep the labeler deterministic and log the file hash. It does not prove the code; it supplies evidence about proof process quality. Evidence: seven annotators labeled 200 states; labels reached Fleiss’ κ = 0.933 and 96.0% raw agreement, while the heuristic achieved 93.8% accuracy and macro-F1 0.933 against 192 consensus states (Section 5.4, p. 9). Limits: the remaining disagreements concentrated on post-implementation read_file calls; in those cases annotators labeled the read Verification while the heuristic labeled it Exploration (Appendix C.5, p. 22), so the heuristic does not always follow the rule above that reads of already edited files are labeled V. Inference: labels depend on the chosen registry and history rules.

technique: Prefix Tree Acceptor reference of valid processes

What it is: merge multiple passing trajectories for one task into a directed acyclic graph that represents alternatives instead of one canonical trace. “Shared prefixes are merged into common nodes, while divergent but successful strategies form branches. Each root-to-terminal path therefore represents one known-good solution” (Section 3.2, p. 5), and the resulting graph encodes a space of correct behaviors. How to do it: require k ≥ 2 passing trajectories for the same task; intent-label each; merge equivalent actions; share common prefixes; branch at valid strategic differences; use an equivalence cascade for surface variants such as grep and rg with the same search intent; score a candidate against this task-specific PTA while excluding that candidate from its own reference. Inference: for VL, build references from successful verified trajectories grouped by repository task, with proof compilation and review events represented as V states. Verus form: Inference: the PTA is an external analysis artifact. A candidate Verus change still needs ordinary requires, ensures, invariant, and proof fn checks; PTA similarity cannot substitute for a proof. Evidence: with k = 2 the pilot AUROC was 0.749 with full task coverage; k = 5 gave AUROC 0.777 and broader solution coverage. At k ≥ 6, the apparent rise came from a small survivor subset, so the authors selected k = 5 (Section 6, pp. 9-10; Table 19, p. 31). Merge-order analysis on one pilot task (astropy__astropy-12907, k = 4) found trajectory selection explained 64.1% of variance and merge order 35.9%; the text says 8 of 10 combinations were order-invariant (Section 6, p. 10), although Table 20 (p. 32) marks three combinations (1, 5, 6) as not order-invariant. Limits: the paper concludes that reference-set choice, not merge order, is the main source of PTA variation (Section 6, p. 10). Inference: a novel but valid strategy may look divergent when absent from the PTA. The paper itself says its Divergent-but-Valid Lucky Passes (C5) “are classified as Lucky only because their solution path diverges from the PTA” (Appendix D.8, p. 30).

technique: composite process-quality score and waste report

What it is: score structural alignment, set coverage, trajectory coherence, and temporal profile. The exact formula is “f(τc, G) = 0.20 · Φstruct + 0.15 · Φcov + 0.30 · (100 · Φcoh ) + 0.35 · (100 · Φtemp ).” (Section 3.3, p. 6) Structural alignment uses ordered recall and unordered precision; coverage is the fraction of matched PTA states; coherence rewards E→I→V and penalizes backtracks; temporal profile compares early, middle, and late stage distributions with Jensen-Shannon divergence (Section 3.3, pp. 5-6). How to do it: compute the four signals against the candidate-excluded PTA; combine with fixed weights calibrated on a disjoint 278-trajectory, 10-task pilot; report per-stage coverage, divergence point, and waste. Waste categories are regression loops, blind retries, redundant steps, unnecessary exploration, and cyclic patterns (Section 3.4, p. 6). Assign passing scores ≥70 Ideal, 47–<70 Solid, and <47 Lucky; failing runs are Partial-fail at ≥47 and Off-track otherwise (Section 3.4, p. 6). Verus form: Inference: treat repeated solver retries, unverified executable changes, and proof edits without rerunning cargo verus as waste signals. A report can require a V event after a contract edit, but only Verus itself establishes the postcondition. Evidence: among 1,136 passing runs, 229 were Ideal (20.2%), 785 Solid (69.1%), and 122 Lucky (10.7%) (Section 5.1, p. 7). Lucky blind-retry instances wasted 11.4 steps each versus 2.7 for Ideal instances, a 4.2× difference (Section 5.2, p. 8). On the 47-task scaled set, the combined score had AUROC 0.766, accuracy 72.0%, F1 0.723, and KS p=0.0017; no individual signal had p<0.05 (Section 5.3, pp. 8-9). Individual trajectory matching scored a higher AUROC (0.805) than AgentLens, which the authors accept because the PTA gives divergence localization and waste attribution (Section 5.3, p. 9). Weights and thresholds were set on the 278-trajectory pilot (Section 3.3, Section 4). These denominators and the 47-task setting matter.

findings, trust, and product threat

The five Lucky categories were Minimal & Unverified (19/122, 15.6%), Brute-Force Convergence (42/122, 34.4%), Incomplete Implementation (41/122, 33.6%), Excessive Exploration (5/122, 4.1%), and Divergent-but-Valid (15/122, 12.3%) (Table 1, p. 7); C2 and C3 together were 68.0% of 122. Model associations were χ²(28)=102.47, p<0.0001 (Section 5.2, p. 8). Lucky rates ranged from 0.5% (Opus 4.5) to 23.2% (GPT-4.1) across the eight model backends (Table 2, p. 8). The scoring pipeline is deterministic, CPU-only, and needs no LLM calls (Sections 1 and 4). The authors state that “process scores should be treated as complementary diagnostics rather than replacements for functional correctness, security review, or human judgment” (Appendix A.3, p. 15), and that a high-scoring trajectory “may still contain an incorrect or insecure patch if the underlying tests or references are incomplete.” The abstract says “We plan to release the project repository soon”, so the dataset and SDK were unreleased when the paper was written. Inference: the analysis trusts tool logs, intent rules, PTA references, thresholds, and the benchmark’s tests, so a process score is evidence for review prioritization, never a proof certificate. Inference: the paper does not show that a Lucky label means a bad patch; its own C5 category is called “the least concerning category”.

Inference: the direct threat to agent-written verified Rust is evaluation blindness: a proof can pass after retries or with an under-specified ensures, while process-only signals may also misclassify a valid novel proof strategy. VL should retain Verus verification as the correctness criterion and use AgentLens-like traces to detect missing verification, repeated solver guessing, and proof/specification divergence. Inference: the paper’s pass-rate filtering problem is already addressed by ordinary proof checking; VL should not redo the PTA algorithm unless it needs process-aware training or triage.

techniques for the catalog

• context-sensitive intent labeling — classify actions from history and edited-file state into exploration, implementation, verification, or orchestration
• PTA process reference — merge multiple passing trajectories into a branched known-good space that tolerates valid alternatives
• composite quality and waste scoring — combine structural, coverage, coherence, and temporal signals while reporting retries, loops, and missing verification Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “The critical design decision is Rule 5: file-inspection commands (grep, cat, ls, git log, find) receive Exploration regardless of when they appear, because these commands read information from the codebase and do not verify the correctness of any prior edit.”): read the whole paper including appendices; checked every quotation, number, section and page reference, and the technique steps against the text; changed the link from a wrong DOI (a different paper) to the arXiv page found in the paper text, moved the Table 1 page to p. 7, corrected the section and page of the grep/pytest example (Section 1, p. 2), added the paper’s own caveats (individual matching AUROC 0.805, C5 classified Lucky only by PTA divergence, labeler reads labeled Exploration, Table 20 versus the 8-of-10 claim, A.3 statement, unreleased tooling), and marked own conclusions “Inference:”; did not check cited prior work, figure contents, or the PDF layout of two-column pages beyond the text extraction, and did not recompute any statistic.

Last edited: 2026-10-17