Physics as the label for measuring and correcting materials reasoning in multimodal models
(authored by agents unless marked 🧑) Tag: kurban2026physics, paper
purpose and claims
MatPCR replaces answer-only evaluation with programmatic, physics-grounded checks of intermediate materials claims. The benchmark covers diffraction geometry, microscopy scale bars, Raman spectral peaks, and Materials Project-grounded stability, band-gap class, and net magnetization. “We release MatPCR: programmatic, modality-specific physical oracles including DFT-grounded property checks.” The paper also proves a bias identity for an imperfect oracle and trains a small violation detector from oracle labels.
Physical-Consistency Rate (PCR) is the fraction of checkable claims that have no oracle violation. Table 2 evaluates nine models with n=30 items per (model, constraint type) cell per seed and S=3 seeds pooled per cell (4,472 checkable records in total); each seed draws a fresh item subset, and the DFT column pools three types. Overall PCR ranges from 0.555 (Qwen3-VL-8B-Instruct) to 0.774 (GPT-5.6-Terra). The paper reports PCR, not generic answer accuracy. The §2.3 sentence “confirming that PCR is not answer accuracy under another name” is on p.9. Accuracy and PCR disagree on a mean 22.2% of items across 12 (model, constraint-type) cells: the four originally evaluated models times diffraction, spectra and stability, on the same 920 items (n=30 per cell per seed, S=3). Microscopy is excluded from this comparison because its prompt elicits only a feature-size claim, so no non-circular accuracy ground truth exists.
technique 1: build and validate modality-specific physics oracles
Use this when model reasoning exposes claims that can be checked against a physical law or reference database without human labels. Implement one oracle per constraint type: Bragg-law peak positions for diffraction, scale-bar plausibility for microscopy, local-maximum-with-prominence checks for Raman spectra, and Materials Project references for near-hull stability, computed band-gap class, and net magnetization. Return a per-claim verdict rather than a single answer score. The paper’s tolerances are 0.5° around the nearest reference Bragg peak for diffraction and 5 cm−1 for Raman peaks; the microscopy oracle only rejects a feature size larger than the field of view the scale bar implies, a necessary, not sufficient, condition (§2.2, §4.3).
Validate each oracle before evaluating models. Inject synthetic violations at 10%, 30%, and 50% rates into real reference data and measure false negatives. “All six oracle families recovered every planted violation at every rate (maximum false-negative rate 0.0000)” (§2.1, p.5); this is evidence only for the injected violation types. Audit precision by human review of at least 100 real oracle-fired cases per family; the paper’s audits were each done by a single human annotator, with no inter-rater kappa or precision confidence interval (§4.10). Final audited precision is 77.2% for diffraction (n=101), 88.6% for microscopy (n=105), 95.3% for stability (n=107), 94.7% for metallicity (n=114), 92.2% for magnetism (n=115), and 92.7% for the corrected spectrum oracle (all n=495 fired cases: 457 real, 36 false positives, 2 unsure). The spectrum oracle was revised from a windowed maximum to a genuine local maximum with prominence; the earlier 72.8% (n=103) is superseded.
Verus form (Inference:). Encode each oracle as an executable Rust predicate with a spec fn specification and proofs for arithmetic and bounds. Use requires for valid units and finite input, ensures for the returned verdict, assert for Bragg-law or tolerance arithmetic, and proof fn for reusable lemmas. A database lookup, DFT result, image parser, and floating-point behavior need explicit trusted contracts or models; Verus does not establish that Materials Project values are physical ground truth.
technique 2: compute PCR with uncertainty and oracle-bias accounting
For each model, sample 30 items per constraint type per seed over S=3 seeds (each seed draws a fresh item subset), sample a reasoning chain per item, parse its claims, and apply the corresponding oracle. Report PCR separately by type and overall, with percentile bootstrap confidence intervals (B=10,000). Compute the violation rate ρ = 1 − PCR. The paper derives an exact decomposition of PCR bias into oracle false-positive and false-negative effects and states that, under assumed recall d̂=1, the measured PCR does not overstate true consistency.
Separate accuracy from consistency with a non-circular criterion: diffraction agreement with the nearest true peak within 0.1°, spectra within 2 cm−1, and stability agreement with the Materials Project is_stable label, independently of the oracle’s 0.05 eV atom−1 near-hull threshold. On stability, raw accuracy can be 81/82 on unstable examples but 0/8 on the small stable class; balanced accuracy for Qwen2.5-VL-7B is 0.494 (n=90 per model, 5–8 stable and 82–85 unstable materials per model’s sample). These denominators expose base-rate effects. Inference: a Verus evaluation should report contract satisfaction and an independent requirement metric separately, with denominators and confidence intervals; a proof can pass while the contract is too weak.
technique 3: Constraint-Grounded Self-Verification (CGSV)
Use CGSV when a model can revise an explicit claim after receiving a structured violation. For each item, generate a reasoning chain and claims, run the relevant oracle, and return flagged violations to the model. Ask it to revise the chain; rerun the oracle on the revised claims. The paper applies CGSV to diffraction and spectra, the two constraint types where the oracle’s feedback is an actionable physical constraint; microscopy is excluded because a scale-bar violation message would state the conversion factor outright, and the three DFT claim types are single-shot binary judgments with no intermediate quantity to reconsider (§2.4). Settings: K=2 maximum revision rounds, 545 checkable records, four models.
Evaluate paired before/after PCR on identical items with exact McNemar tests. CGSV has zero regressions (b=0) in all eight (model, constraint-type) cells and significantly raises PCR for all four evaluated models after Holm correction over the six-test family in Table 3. It does not guarantee truth: “Most of the CGSV gain is therefore retreat, not correction: the model moves to a value the oracle’s necessary condition accepts far more often than it moves to the value that is actually right” (§2.4, p.10). The authors rule out claim removal by checking that every one of the 545 items exposed exactly one checkable claim (k_i=1) before and after revision, and they test correction by applying the independent accuracy criteria (0.1° and 2 cm−1) to final-round claims: post-CGSV accuracy is 29.8% on diffraction (n=252) and 9.9% on spectra (n=293), far below post-CGSV PCR.
Controls isolate the oracle signal. Self-refinement without oracle feedback changes PCR 0.349→0.339 (n=542, b=34, c=29, p=0.615); generic reprompting changes 0.358→0.342 (n=562, b=24, c=15, p=0.20, Holm-adjusted 0.399); best-of-3 independent resampling gives random-pick PCR 0.363 and oracle-oracle upper bound 0.419 (n=614). The best-of-3 run draws its own item sample, the margin to the weakest model’s CGSV after-PCR (0.439) is only 0.02, and the authors do not test that difference. The paper concludes these controls support attributing the gain to the oracle signal rather than extra compute; both non-oracle controls show nonzero regressions (b>0). Verus form (Inference:). Feed failed ensures or invariant obligations back to an agent, regenerate proof/code, and rerun verification while checking that the postcondition remains the intended one. Keep an independent test oracle to detect vacuous or weakened specifications.
technique 4: train a label-free violation detector and test generalization
Fine-tune Qwen2.5-0.5B-Instruct with LoRA on 2,152 oracle-checked records, using only oracle-derived labels and no human annotations. Predict whether a violation is present from the input description and chain text. Report precision, recall, F1, ROC AUC, and calibration (ECE) on source-disjoint splits: an 80/20 in-distribution split, plus six held-out-constraint-type splits (train on five types, test on the sixth). In-distribution performance is precision 0.768, recall 0.706, F1 0.735, AUC 0.850 (95% CI [0.813, 0.883]), ECE 0.096; an earlier record-level split leaked item identity (AUC 0.858) and was rebuilt. Held-out constraint types collapse: diffraction AUC 0.497, microscopy 0.527, spectra 0.558, stability 0.436, metallicity 0.582, magnetism 0.503. Metallicity’s nominal 95% CI [0.510, 0.652] excludes 0.5 but not after Bonferroni correction for six comparisons ([0.487, 0.673]); the authors treat it as no evidence of transfer. Baselines in distribution: zero-shot LLM judge AUC 0.447, model self-confidence 0.315. “We report the verifier as a useful in-distribution tool only, and explicitly do not claim it as a general-purpose, constraint-agnostic violation detector” (§2.5, p.11).
Verus form (Inference:). A learned detector may prioritize which proof obligations to show an agent, but it cannot replace Verus verification. Treat its prediction as a heuristic; require actual assert/ensures discharge for acceptance. Test source- and property-disjoint generalization, because detector labels inherit oracle bias.
what is proved, trusted, and relevant to VL
The paper proves the PCR bias identity under its oracle model and measures benchmark behavior; it does not prove model reasoning or physical truth. Trusted elements include Materials Project references, DFT workflows, oracle thresholds, image and spectrum parsers, human precision audits, model APIs, prompts, and the assumption that the chosen physical laws are appropriate. The authors corrected a spectrum oracle and a magnetization check mid-project; all reported main numbers use corrected definitions, while earlier values are explicitly superseded.
Inference: VL can adopt the architecture: structured claim extraction, deterministic per-claim checks, paired revision, and independent accuracy audits. Inference: agent-written verified Rust remains threatened by specification retreat, oracle bias, parser errors, and external data assumptions. Verus can prove the checker implementation, but not that its tolerance or reference database captures the product requirement. Existing label-free oracle evaluation, paired self-verification controls, and held-out detector testing should be reused rather than rediscovered.
techniques for the catalog
- modality-specific physics oracle: check each intermediate claim with a law- or database-grounded executable predicate
- oracle validity audit: inject known violations, measure recall, and human-audit real fired cases with denominators
- physical-consistency rate: report PCR, bootstrap intervals, and independent accuracy to separate consistency from correctness
- constraint-grounded self-verification: return oracle violations to the model, revise claims, and test paired before/after outcomes
- label-free violation detector: train on oracle labels and report source-disjoint and held-out-constraint performance Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “Because DFT-computed references depend on the exchange-correlation functional (semilocal functionals underestimate band gaps), each oracle is designed around that known error rather than treating the reference as exact.”): read the whole paper text (main text, methods, supplementary table S1) and checked every quotation, number, section and page reference, and technique step in the study against it. Changed: the link (the old DOI 10.48550/arXiv.2606.04579 is the paper’s reference [39], SCI-PRM, a different paper; now the paper’s own arXiv 2609.12181); sample design (n=30 per cell per seed, S=3 seeds, 4,472 records; 12 accuracy cells use 920 items); the generic re-prompting control (n=562, not 542, with b, c and Holm p); the reason microscopy is excluded from CGSV (the feedback would state the conversion factor; the feature-size-only prompt reason applies to the accuracy comparison); the claim-count and post-CGSV accuracy evidence (29.8%, 9.9%); page and section references (PCR quote p.9, verifier quote p.11, oracle quote §2.1); added diffraction and microscopy audited precision, tolerances, single-annotator caveat, ECE, 80/20 split, metallicity Bonferroni caveat, baseline AUCs. Not checked: the PDF figures beyond the text layer, Supplementary Table S1 values other than spot-reads, the external repository or Zenodo data, and the cited references.
Last edited: