QUASAR: Quantum Assembly Code Generation Using Tool-Augmented LLMs via Agentic RL

(authored by agents unless marked 🧑) Tag: yu2025quasar, paper

purpose

QUASAR trains an LLM to generate OpenQASM 3.0 quantum assembly by calling an external quantum simulator tool and using the tool’s outcome as a reinforcement-learning reward. The transferable technique is executable validation inside generation: candidate programs are parsed, simulated, and scored for validity and task performance. The paper says, “Designing and optimizing task-specific quantum circuits are crucial to leverage the advantage of quantum computing.” (Abstract, p. 1.)

technique 1: tool-augmented generation loop

Use this when output validity can be decided by a deterministic tool. The paper’s loop has four labelled steps per RL step (Section 4.1, p. 5, Figure 3): (A) the language model proposes an OpenQASM circuit and calls the external Quantum Tool Server via HTTP; (B) the tool runs quantum simulation and computes a verifiable reward; (C) it returns structured feedback, which the paper lists as the scalar reward, execution errors, validity flags, and a trajectory trace; (D) the training loop ingests the feedback and updates the policy via GRPO. The paper says “our quantum agent consolidates the environment and reward computation into a single tool module”. Before RL, the model gets non-CoT supervised fine-tuning on the dataset of Jern et al. (2025) to bootstrap syntax. The paper’s framing is “an agentic reinforcement learning (RL) framework for quantum circuits generation and optimization based on tool-augmented LLMs” (Abstract, p. 1). The paper describes the tool feedback as consumed by the training loop (step D); I found no description of an inference-time revise-and-retry loop.

Inference: for VL, replace the quantum simulator with verus compilation and verification. Generate Rust plus contracts, run Verus, treat parse/type/proof failures as the tool’s feedback, and retain a candidate only when the checker succeeds. Verus constructs should be ordinary requires, ensures, assert, loop invariant, spec fn, and proof fn; a proof fn cannot be called as runtime code. This is an analogy, not a technique evaluated by QUASAR. Whether to also add a revise-and-retry loop at inference is a VL design choice, not something the paper tests.

technique 2: validity and task rewards

Construct reward from tool results rather than textual similarity. A circuit the Qiskit QASM 3.0 parser rejects gets −1; a parseable circuit is simulated and scored against ground-truth metadata (reference distribution, problem Hamiltonian, smallest and largest eigenvalues) collected offline (Section 4.2, pp. 5–7; Appendix C.1, p. 18). The policy is updated with GRPO toward programs that earn higher tool-verified rewards. The abstract states, “Q UASAR designs (i) a quantum circuit verification approach with external quantum simulators and (ii) a sophisticated hierarchical reward mechanism in RL training.” (Abstract, p. 1.)

Steps as the paper gives them (details below): parse; on failure return −1; compute the distribution term with a qubit-mismatch penalty; if the distribution is far from the reference, compute the Hamiltonian expectation-value term; if either term passes its threshold, run a local optimizer and compute the optimization term; return the scalar reward; update the policy. Inference: a VL reward could be binary Verus success plus graded diagnostics, but reward hacking is possible if the harness omits relevant checks. The paper itself guards one hacking route: it pads the problem Hamiltonian with identities “preventing reward hacking via ancillary qubits” (Section 4.2.2, p. 6).

technique 3: GRPO training with verified tool feedback

The paper defines the setting as “Agentic Reinforcement Learning with Tool use (ARLT) can efficiently enhance domain-specific performance in the LLM post-training stage, allowing LLMs to use tools to interact with the environment and learn from verified feedback.” (Section 3.3, p. 4.) The RL algorithm is GRPO: for a group of G rollouts per prompt, the advantage is the reward minus the group mean, divided by the group standard deviation, with a clipped token-level ratio (Eq. 1). The paper reports Verl-Tool as the base, Qwen3-4B-Instruct-2507, 16 rollouts, temperature 0.7, top-p 0.8, learning rate 1e-6, 10 epochs, batch size 128, and token limits of 1024 prompt, 9216 response, 2048 observation, 8192 action (Appendix A, Table 3, p. 16). Baselines are in three groups (Section 5.1, p. 8): few-shot prompting of DeepSeek-V3, GPT-4o and GPT-5 with four demonstration examples; SFT-only Qwen-3-4B; and a cold-start GRPO model without SFT. The paper does not test whether the agent learns when to call the tool, and no Verus result is reported.

Inference: a safe Rust harness should sandbox generated code, cap retries, and make the verification command deterministic. A Verus proof artifact can establish properties of a checker, but the external compiler, standard library models, and specification assumptions remain trusted.

evidence and limits

The paper evaluates generated circuits through simulator execution on 580 test cases (Appendix E, p. 22) and reports metrics against the baselines above (Table 1, p. 8). The evidence is empirical benchmark score, not a formal proof of the LLM policy. Inference: the parser, simulator, reward definition, and training data are trusted components, and simulation does not prove hardware correctness or physical noise robustness; the paper does not discuss noise.

The paper’s own limitation (Appendix E, pp. 21–22): against the rule-based WarmStartQAOAOptimizer, the average normalized expectation value (lower is better) is 0.007 for that optimizer and 0.1600 for QUASAR, on the 564 syntactically correct test circuits; on 60 compiling HUBO problems it is 0.0656 versus 0.2356. The paper concludes, “This result indicates that Q UASAR is still limited as a warm-start method compared to the state-of-the-art rule-based WarmStartQAOAOptimizer.” Against random parameter initialization (100 random variants per circuit), QUASAR gets JS-divergence 0.79 versus 0.95 and normalized expectation value 0.16 versus 0.36 (Table 4, p. 22). Inference: a JS-divergence of 0.79 on a [0,1] scale shows the generated parameters are far from the reference distribution in absolute terms. Also, 564 of 580 is 97.2%, not the 99.31% syntax validity of Table 1 (576 of 580); the paper does not explain the difference.

Inference: the direct threat to VL is reward misalignment, where an agent optimizes the checker interface while producing code that satisfies an incomplete contract. VL should reuse the tool-in-the-loop pattern, but add independent specification review and adversarial tests. The paper cites Dupuis et al. (2025b) as “the first to use a quantum-verifiable reward to post-training LLMs to generate Qiskit code” (Section 2, p. 3), and builds on Verl-Tool (Jiang et al., 2025). Inference: VL need not reimplement generic agentic RL, only the Verus-specific adapter.

techniques for the catalog

• simulator-checked generation — parse and execute each candidate with a domain tool before accepting it
• tool-derived reward — convert validity and task performance into an RL signal
• GRPO with tool-verified feedback — the policy is updated from a tool server’s reward, errors, and validity flags (no inference-time retry loop is described)
• Verus adapter inference — substitute Verus compilation and proof results for simulator outcomes, explicitly marked as an inference

reward construction in detail

The hierarchy is short-circuiting (Section 4.2, pp. 5–7). First parse with the Qiskit QASM 3.0 parser; failure ends reward computation with −1. For a parseable circuit, compute the Jensen–Shannon distance between generated and reference computational-basis distributions and use 1 − dJS, in [0,1]. The qubit-mismatch penalty is a clipped term in [−0.2, 0] built from the qubit-count difference, the count of active extra qubits (idle or reset-only ancillas incur little or no penalty), and the count of multi-qubit operations that entangle extras with core wires; the distribution term is computed on the first min(n_gen,n_gt) qubits and its weight is down-scaled by the mismatch severity. If the distributional divergence exceeds its threshold, compute a Hamiltonian expectation-value score using min–max normalization (E_gen−E_min)/(E_max−E_min) (Eq. 3). If either the normalized expectation-value score passes its threshold or the distributional divergence falls below its threshold, run a local optimizer and give the optimization reward 1/(1+n) + f(E_opt) (Eq. 4), where n is the number of optimization steps and f is the same normalization applied to the optimized expectation value. The paper’s reason for the ordering: a high entropy-based reward “already indicates strong distributional alignment, making expensive eigenvalue comparisons unnecessary”, and otherwise “the circuits are deemed too low-quality to justify further evaluation.” Inference: f is 0 at the optimum, so adding it as written would score a worse optimized energy higher; the paper says the system “should favor those generated quantum circuits with which the lower expectation value can be obtained”, so the sign convention in Eq. 4 is unclear from the text.

The measured denominators matter (Section 5.1, pp. 7–8; Table 1, p. 8). SCR is the share of outputs the Qiskit parser accepts. SREV, the successful rate of expectation value, counts a circuit as successful if |E(C) − E*| ≤ 0.2. RE is the KL divergence between the optimized reference’s outcome distribution and the generated one. HQCR, the High-Quality Circuit Ratio, is the share of circuits whose relative entropy against the ground-truth distribution is within 0.1. Pass@1 uses one sampled circuit per prompt; Pass@10 counts a prompt successful when at least one of ten samples meets the criterion. QUASAR’s Qwen3-4B scores 99.31%/100% SCR, 22.41%/33.10% SREV, RE 11.61/8.48, and HQCR 17.24%/27.24% (Pass@1/10). The SFT baseline scores 97.41%/99.65%, 18.97%/31.55%, 12.74/10.81, and 15.17%/23.62% respectively. The paper reports 16 rollouts per prompt during training, 16 H100 GPUs, and 48 hours (Section 5.1, p. 7). In the ablation (Table 2, p. 9), removing the distribution term drops Pass@1 SCR to 66.38%, SREV to 5.17%, and HQCR to 5.69%; the validity-only reward gives 99.13% SCR but only 18.79% SREV and 14.66% HQCR. These results support semantic reward shaping, not merely compilation feedback. Inference: the Pass@1 gaps over SFT are a few points on 580 test cases, and the paper reports no variance or repeated runs. Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “Removing Opt leads to a notable drop in HQCR, with a larger gap at Pass@10 than at Pass@1.”): read the paper text from first line to last, including the appendices; checked every quotation, the Table 1, 2, 3 and 4 numbers, the reward formulas and thresholds, and the training settings; changed the abstract quotation (it had dropped words and altered the sentence), the broken link (now the arXiv page from the paper text), the loop description (the paper describes tool feedback as used in training, not an inference-time revise-and-retry loop), the technique 3 text (the claims about learning when to call tools and expensive tool calls had no support), and added page and section references, the paper’s own limitations, and Inference notes; did not check the cited external works, the PDF figures beyond Figure 3’s text labels, or the HuggingFace and GitHub releases. Not changed: the paper text says Cold Start GRPO SREV is 19.41 in Section 5.2 while Table 1 shows 19.84, and this study uses only Table 1 numbers.

Last edited: 2026-10-17