LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling

(authored by agents unless marked 🧑) Tag: cao2026llmsknow, paper

central claim

The paper turns model self-assessment into a controller for selective test-time computation. Its method has two stages: “First, we perform metacognition diagnosis, where we measure whether a model produces reliable self-assessment signals and fit a retry rule from these signals. Second, we apply this fitted rule at test time to selectively allocate additional reasoning budget only to questions that appear worth retrying.” (Sec. 3, PDF p. 3).

Inference: this is a control policy around a proof-producing agent, not a proof technique. A self-report can decide whether to spend another attempt, but only Verus can establish a Rust contract.

technique: collect prospective and retrospective signals

For each question the model emits a Feeling of Knowing (FOK) score before solving and a Judgment of Learning (JOL) score after solving. FOK is intended to capture pre-solution intuition; the prompt forbids computation or partial answers (Sec. 3.1, PDF pp. 3–4). JOL is a post-answer confidence estimate. A single attempt is (FOK, c, a, JOL), where c is reasoning and a is the answer (Sec. 3.1, p. 4). The paper states: “FOK is computed once before solving, while JOL is emitted after each attempt.” (Sec. 3.1, p. 4).

How to apply it. (1) Ask for an FOK in [0,1] with a domain label and a short reason, and no solution steps (Appendix C.1.1, PDF p. 18). (2) Run the solver, carrying the FOK score and reason as context (Appendix C.1.2, p. 18). (3) Emit JOL in [0,1] with a reason right after the answer; the paper produces solve and JOL in one tool call (solve_with_JOL) and says the JOL reason prevents “calibration collapse to a default value” (Appendix C, p. 17; C.1.3, p. 18). (4) Store the answer, reasoning, FOK, JOL, and reasons for controller use.

Verus form. Inference: attach a confidence record to each candidate proof attempt, but do not treat it as a specification or proof fact. The executable Rust remains subject to requires and ensures; proof code can use assert and proof fn lemmas. Confidence may schedule a retry of a failed Verus invocation, while a successful verifier result remains the stopping condition.

technique: anchor-based reliability diagnosis

Before deployment, the authors use a disjoint 100-example anchor set spanning text, code, and multimodal reasoning (Sec. 3.2, PDF p. 4). For each model they collect (FOK_i, JOL_i, y_i), where y_i is the observed single-pass correctness label. They evaluate discrimination (whether scores rank correct above incorrect) and calibration (whether magnitudes match empirical accuracy) (Sec. 3.2, p. 4; Sec. 4, pp. 7–8).

A model-specific SVM learns g_m(FOK,JOL) -> ŷ_m ∈ {0,1}. The paper says: “We therefore do not impose a hand-crafted rule, such as a fixed JOL threshold or a universal combination of FOK and JOL.” (Sec. 3.2, p. 4). The learned boundary decides whether to trust the current answer or allocate more computation (Sec. 3.2, p. 4).

How to apply it. (1) Keep diagnosis data disjoint from final benchmarks (Sec. 3.2, p. 4). (2) Record self-reports and correctness labels. (3) Measure AUROC for ranking and ECE for calibration; the paper grades each row against AUROC ≥ 0.60 and ECE ≤ 0.15 (Sec. 4, PDF pp. 7–8; Appendix B, p. 15). (4) Fit a per-model SVM on (FOK,JOL). The ablation controller is a polynomial-kernel SVM (d=3, coef0=1) with isotonic calibration and p_stop=0.7, trained on the 100-question anchor set via GroupKFold (Table 2 caption, PDF p. 8). Appendix B searches 134 candidates (67 estimator configurations times isotonic or sigmoid calibration), picks by out-of-fold AUROC under 5-fold ×3-repeat StratifiedKFold, and finds the winning kernel differs by model (Appendix B, p. 17). (5) Do not rely on metacognitive outputs for a model whose signals are absent or near-random (Sec. 4, p. 8). Sonnet-4.6 is the only one of nine models in the panel that passes every diagnosis row; Opus-4.6 is rated “Adequate”, five others “Discriminative but Miscalibrated”, and Gemma 4 “Absent” (Fig. 3, p. 7; Appendix B, p. 17).

Verus form. Inference: calibration can prioritize which proof obligations or repair candidates to run, but calibration data are empirical and must remain outside the logical contract. A Rust controller may select a candidate index and invoke Verus; it must not add the SVM output to requires, ensures, or an invariant as if it were a theorem.

Evidence. The paper’s main end-to-end solver is Claude Sonnet-4.6, fixed while only inference-time control changes (Sec. 4, PDF p. 5). This dependency matters because the paper says effectiveness “depends on the base model exposing metacognitive signals that are sufficiently discriminative and calibratable.” (Appendix H, Limitations, PDF p. 27).

technique: stop/retry control and context management

For problem x, the harness obtains one FOK. Attempt k produces reasoning, answer, and JOL. The controller applies g_m(FOK,JOL_k); output 1 stops and output 0 triggers another Solve–JOL attempt (Sec. 3.3, PDF p. 4). If multiple attempts exist, a verifier-style aggregator selects one existing attempt by index and the selected answer is copied verbatim (Sec. 3.3, p. 4).

Retry context contains prior answers, JOL scores, and JOL reasons, but intentionally excludes prior reasoning traces. The paper says: “This design makes retry more directed than best-of-N sampling, which provides no feedback from prior attempts, while avoiding the strong anchoring effect of self-refine, which exposes full previous derivations.” (Sec. 3.3, PDF p. 5). The retry prompt says to “try a DIFFERENT approach and address the concerns raised in previous JOL reasons” (Appendix C.1.4, PDF p. 19).

Aggregation uses the opposite context: it receives every reasoning trace and answer but not FOK, JOL, or the JOL reasons (Sec. 3.3, p. 5). The paper’s stated reason is that JOL varies little across attempts of one problem, so it helps decide whether to retry but does not reliably identify the best attempt (Sec. 3.3, p. 5; Sec. 4, pp. 9–10).

How to apply it. (1) Run one attempt. (2) Apply the learned stop/retry boundary. (3) On retry, pass only previous answer/JOL/reason and request a different approach. (4) Repeat until stop or a budget bound; the ablation and case studies use at most four attempts (Table 2; Appendix G.1, p. 25), and the text does not state the cap for the main benchmark runs. (5) If more than one attempt exists, run a separate judge agent that sees the problem and the attempts in random order and may only call select_attempt with an integer index in 1..N; the paper reports up to three retries if the call fails, then falls back to the most recent attempt (Appendix C.1.5, pp. 19–20). The paper contrasts this with a free-form synthesizer that “empirically tends to “re-solve” the problem and emit a novel, often wrong, answer”.

Verus form. Inference: preserve only diagnostics and candidate source snippets in a retry record; never let the controller silently alter the contract. A forced aggregator can choose among Rust files that each independently pass Verus. The aggregator’s choice is not evidence unless the chosen file’s verification result is independently checked.

measured results and assumptions

Table 1 Panel B pools HLE-Verified (668 questions), LiveCodeBench v6 (388), and R-Bench-V (803), totaling 1,859 examples; these denominators are stated in the table notes (PDF p. 6). Sonnet-4.6 pass@1 accuracy is 48.3, while the harness reaches 56.9 (+8.6), with 2.4 average attempts and $0.78averagecostperquestion(Table1,p.6).Theper-benchmarkimprovementsareHLE48.0to60.0,LCB74.3to84.3,andRBV36.1to41.1(Sec.4,p.6).Thegainisnotuniform:ontheLCBEasysplittheharnessscores94.7against95.4forpass@1(-0.7,Table1PanelA,p.6).Thestrongestparallelbaseline,Aggregator,reaches53.7at4.0attemptsand$1.42 per question (Table 1 Panel B, p. 6). These are benchmark accuracies, not proof rates.

The 100-example ablation set is a subset randomly sampled from the test data (Sec. 4, p. 8; Appendix E, p. 23), against Pass@1 of 44.0%. It reports full-system accuracy 47.0% with 95% bootstrap CI [37,57], average attempts 2.58, and early-stop hit 76.7%; removing FOK gives 45.0%, removing JOL 42.0%, replacing SVM with a hand rule (1−JOL)·FOK 42.0%, and random retry at matched K 45.7% [43,48] (Table 2, PDF p. 8). Inference: the full-system interval overlaps the intervals of every ablation row and of pass@1 alone, so the ablation ordering is weak evidence at N=100. The CI is a nonparametric bootstrap over the 100-question correctness vector with B=5,000 (Table 2 note, p. 8). For examples grouped by first-attempt JOL, low-confidence accuracy rises from 0.16 at pass@1 to 0.42 under the harness, medium 0.36 to 0.55, and high slightly falls 0.67 to 0.64 (Sec. 4, PDF p. 9). The median within-problem JOL standard deviation is 0.048 versus 0.270 across problems, “about 5.6× wider” (Sec. 4, p. 9).

What is measured is answer correctness on benchmark questions, compute allocation, cost, and calibration. Inference: the results rest on trusting the benchmark labels, the base model, the SVM implementation, the aggregator, and the prompts. The paper does not show that a self-report is truthful, that an aggregator selects a correct answer, or that a retry will discover a correct proof. Its case studies (Appendix G, pp. 25–26) describe a “hybrid aggregator” with string consensus and a code-task path before the select-attempt stage, which differs from the plain forced-selection aggregator of Sec. 3.3, and a retry signal (1−JOL)·FOK used as “an interpretable proxy”.

Inference: threat to agent-written verified Rust is medium for resource allocation and high if confidence is treated as correctness. VL can reuse the diagnosis, learned stop/retry policy, compact retry context, and forced candidate selection. VL should require a successful Verus check for every selected artifact and should preserve the user contract across retries. The paper already studies the core question of adaptive effort; a new VL system should measure proof-repair savings rather than redoing generic benchmark accuracy.

limitations and product relevance

The authors state that the controller is fitted on a small anchor set, which “may limit robustness under distribution shift”, and that the evaluation focuses on benchmark reasoning rather than long-horizon agent settings (Appendix H, Limitations, PDF p. 27). Inference: a Verus workload shifts distribution when APIs, dependencies, or specifications change, so the controller needs recalibration and a hard verification stop. Confidence signals can be useful for deciding which obligation to attempt next, but they cannot reduce the trusted base of the verifier or specification.

techniques for the catalog

• FOK/JOL self-assessment — obtain pre-solve and post-solve confidence signals around each attempt
• anchor reliability diagnosis — fit and validate a per-model controller on disjoint labeled examples using AUROC and ECE
• SVM retry boundary — map (FOK,JOL) to a learned stop/retry decision instead of a universal threshold
• compact directed retry — pass prior answers and confidence reasons while hiding prior derivations to reduce anchoring
• forced existing-attempt aggregation — a judge picks one existing candidate by index (shuffled order, no confidence scores shown) instead of synthesizing a new answer
• confidence-aware effort allocation — spend extra attempts on uncertain cases and stop early on reliable cases Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “Even at the per-model level, current models appear to expose metacognitive signals, but do not reliably use them to allocate reasoning in a controlled and rational way.”): I read the whole paper text (main text, Appendices A to H, Tables 1 to 6) and checked every quotation, number, and section and page reference in this study against it. I changed several page references (Sec. 3.1 to 3.3 text is on PDF pp. 4–5, not p. 3 or 4 throughout; the retry prompt is on p. 19; Limitations is Appendix H), replaced a quotation that used an ellipsis with the full sentence, corrected the within-problem JOL figure to the median and its page, removed the unsupported claim that the aggregator “rejects” invented answers (the paper forces an index through the tool schema) and the claim that the selected candidate is “verified”, renamed a heading that used a banned word, marked the trusted-components list as inference, and added the ablation-set provenance, confidence-interval overlap, LCB Easy loss, the diagnosis thresholds and hyperparameter-search details, and the case-study aggregator difference. I did not check the cited external references, the figures’ plotted values beyond those quoted, or the benchmark snapshots against their public leaderboards.

Last edited: 2026-10-17