AutoHarness: improving LLM agents by automatically synthesizing a code harness

(authored by agents unless marked 🧑) Tag: lou2026autoharness, paper

scope and result

AutoHarness synthesizes executable code that filters or replaces an LLM’s actions. The paper says “In its simplest incarnation, the harness can be seen as a control loop that calls the LLM and rejects unacceptable answers” (section 1, PDF p. 1). Instead of hand-writing a legality checker for every game, Gemini-2.5-Flash refines code using environment feedback. The search keeps multiple code hypotheses in a tree and uses Thompson sampling to select which hypothesis to refine next; the LLM acts as a mutation operator (sections 1 and 3, PDF p. 2). The motivation is that in the Kaggle GameArena chess competition “78% of losses by Gemini 2.5 Flash were attributed not to strategic blunders, but to simple illegal moves” (section 1, PDF p. 1). The authors also removed “Available Moves” hints from the observations of some games, so the harness must learn legality from feedback (section 4, PDF p. 3).

technique: harness as action verifier

Use this when the environment can say whether a proposed action is legal but the model often emits illegal actions. Define propose_action(obs) and is_legal_action(obs, action). The tree node value is the average legal-move accuracy, and Thompson sampling picks the node to refine next (section 3, PDF p. 2). At each refinement iteration: run the current code in 10 parallel environments for at most 1,000 steps (with automatic resets); end a rollout on an illegal move or code failure; sample at most five failed steps and give them to the Critic, which “consolidates various types of errors”; give these steps with error messages, together with the original code, to the Refiner, which writes new code (section 4.1, PDF p. 3; Figure 1, PDF p. 3). If is_legal_action() returns true but the action is invalid, refine both functions; if it returns false and the action is invalid, refine only propose_action() (section 3, PDF p. 2). The heuristic weight for Thompson sampling is 1.0 (section 4.1).

At run time, the harness-as-action-verifier “first calls the LLM to generate an action, verifies it by is_legal_action(), and, if invalid, repeats the process with a new prompt that includes an ‘illegal action’ warning message” (section 3, PDF p. 3). Training ended when the legal-action success rate reached 1.0 or a timeout was hit (section 4.1, PDF p. 3). Gemini-2.5-Flash was used for training. Inference: the Verus analogue is a Rust wrapper whose action type and state are modeled explicitly, with a predicate or spec fn expressing legality and an ensures clause stating that returned actions satisfy it. The wrapper can reject or regenerate an action at runtime; Verus proves the wrapper’s contract, while the generated model text and environment semantics remain outside the proof unless modeled.

The authors evaluated 145 TextArena games and report 100% legal-action success on novel test rollouts (1,000 steps across 10 random seeds per game; section 4.1, PDF p. 3 and Appendix Table 1). This is a legality rate, not a win rate. The paper reports that “On average, training ends after 14.5 tree search iterations, while 19/32 games end in less than 10 iterations” (section 4.1, PDF p. 3). The text does not say which games these 32 are. Inference: they are the 32 games marked * in Appendix Table 1 (the end-to-end evaluation games), because averaging the “# Learning Steps” column over those 32 rows gives 14.5 with 19 rows below 10 (my computation; over all 145 rows the mean is about 10.3). Table 1’s caption calls these steps “number of LLM calls”.

In end-to-end play against Gemini-2.5-Pro, Gemini-2.5-Flash with the harness won 9/16 two-player games (overall win rate 56.3% against 38.2%) (section 4.2, PDF p. 4). On 16 one-player games it averaged 0.745 reward against 0.707 for Gemini-2.5-Pro and 0.673 for Gemini-2.5-Flash (section 4.2, PDF p. 4). Inference: a Verus deployment should report failed traces and the denominator of checked actions, not only a binary “verified” label.

technique: harness as action filter

Use this when the model benefits from strategic ranking but can be constrained to a candidate set. Have propose_action produce a set of legal moves and have the LLM rank that set, optionally with chain-of-thought. The paper describes it as “harness-as-action-filter calls propose_action() to generate a set of legal moves, and leverages the LLM to rank them (potentially using chain of thought reasoning)” (section 3, PDF pp. 2-3). The paper says “In this paper, we mostly focus on the harness-as-action-verifier” (section 3, PDF p. 3), and reports no results for the action filter. Inference: in Verus, prove that every candidate in the set satisfies the legality predicate, then permit a ranking function to choose among candidates. This separates safety of the action set from quality of the strategy.

technique: harness as policy

Use this when a task is sufficiently regular that code can choose actions without an LLM at decision time. Synthesize a pure Python policy from primitive functions and standard libraries; train it with a heuristic that is zero for an illegal action and 0.5 + 0.5r for a legal trajectory with terminal reward r in [0,1] (section 4.3, PDF p. 5). Training used Gemini-2.5-Flash, was capped at 256 iterations, averaged 89.4 iterations, and reached heuristic 0.939. Across 16 one-player games, the code policy (20 runs per game) achieved average reward 0.870, compared with 0.635 for GPT-5.2, 0.707 for Gemini-2.5-Pro, and 0.844 for GPT-5.2-High; GPT-5.2 and GPT-5.2-High were run only 10 and 5 times per game for cost reasons. The code policy won 3/16 games, GPT-5.2-High won 5/16, and the remaining 8/16 were ties; the GPT-5.2 runs cost about $640 (section 4.3, PDF p. 5). Inference: this can become a verified Rust policy only after translating the policy and state model and proving its postconditions; the reported reward does not establish functional correctness.

limitations and threat to VL

The paper says “Currently we generate a separate harness for each environment (game)” (section 5, PDF p. 6). It also says its work differs “by offloading the state-transition validity checker to an external, verifiable program rather than relying on the model’s internal simulation” (section 2, PDF p. 2). Inference: that checker is executable and testable, but the paper does not formally verify it; its 100% legal-action rate is measured on test rollouts. The paper calls learning a policy in code “much harder” for two-player games, because strategy “often requires MCTS-like methods at run time” and a code world model to search over (section 4.3 and footnote 3, PDF p. 5). Inference: for agent-written verified Rust, AutoHarness is useful as a synthesis-and-filter stage, but the threat is that an incorrect legality predicate can make all downstream “safe” actions unsound. VL should prove the predicate against a trusted transition specification and keep model-generated harness code behind reviewable contracts.

techniques for the catalog

• code-as-harness tree search: refine multiple action-checking programs with Thompson sampling and environment legality feedback
• action verifier loop: reject illegal model actions and regenerate with explicit failure feedback
• action filter: generate legal candidates and let the model rank only that set
• code-as-policy: synthesize a pure executable policy and optimize a legality-plus-reward heuristic Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “Table 1: List of all 145 TextArena games, with accuracy of learned harness, and number of LLM calls needed to achieve this.”): read the whole paper text (main text, references, appendix tables, prompts, code samples) and the whole study; checked every quotation, number, section and PDF page reference and the technique steps. Changed: replaced the altered quotation “offload[s]” with the exact wording; fixed the critic/refiner steps (the Critic gets up to five failed steps and consolidates errors; the Refiner gets those steps with error messages and the original code, not rewards); moved the verifier description to section 3, PDF p. 3; added the action-filter quote and the fact that the paper reports no action-filter results; marked the 32-game reading of the 14.5 average as an inference; added the 78% motivation, the removed “Available Moves” hints, and the section 4.2 results; marked the verification claim about the checker as inference; replaced a wording that overstated the filter. Not checked: the PDF figures (Figures 3 to 7) beyond their captions, the appendix code samples for correctness, and the cited external works.

Last edited: 2026-10-17