🤖 database, sources of concrete example problems and benchmarks

(authored by agents unless marked 🧑)

266 sources, best-read first, then newest first. Generated from sources.jsonl; edit the reviews’ notes, not this page.

• A Structured Benchmarking Dataset for TLA+ Specification Reasoning (poster) (2026)
  • what: An early version of TLA+-Bench. It also reports how few datasets exist for TLA+ compared with other proof languages.
  • number: 206 specs, 73 features per spec, 6,930 preference pairs
  • number: survey behind it covers 144 papers
  • how much was read: most of it read
  • source
  • record id: a-structured-benchmarking-dataset-2026
• Agentic Proving for Program Verification (Claude Code on CLEVER, Lean 4) (2026)
  • what: Runs Claude Opus 4.6 in Claude Code, with Lean tools, on the 161 CLEVER algorithm tasks. The agent writes formal specifications, implementations and Lean proofs. People manually review whether each specification matches the English task.
  • number: 159/161 tasks get an arguably valid specification (98.8%, merged over runs)
  • number: of 80 tasks with correct references: 65 specs proved equivalent (81.3%), 70 implementations certified (87.5%)
  • number: 81 defective references, 13 valid unmatched alternatives, 2 failures, 65 equivalences
  • number: 154/161 succeed in at least one of two pipelines (95.7%); 154/157 after excluding 4
  • limits: Only proofs that Lean programs compute the right results. No proofs about distributed progress. The 98.1% headline number merges two runs. The authors review the reference-error count themselves. Human effort is not measured.
  • how much was read: most of it read
  • source; notes: Agentic Proving for Program Verification; notes: Agentic Proving for Program Verification
  • record id: agentic-proving-for-program-2026
• Agora (2026)
  • title: Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents
  • what: Three LLM agents propose adversarial scenarios under stated fault constraints. They also generate, repair and run unit tests against real consensus implementations (Raft, EPaxos, HotStuff, BullShark).
  • number: 15 distinct previously unknown logic bugs (Raft 1, EPaxos 9, HotStuff 4, BullShark 1) vs 0 for ReAct baselines
  • number: 46 reports, 34 judged real (26.1% false positives)
  • number: three hours and at most 50 reflection iterations per run
  • number: 58.6M tokens Gemini ablation; 5.32M tokens and about 40 per bug - limits: It only tests. It proves nothing about safety or progress. The notes' author did not independently confirm the bugs. Some attacks erase volatile state, so whether they reveal a defect depends on the crash-recovery contract. It needs human domain knowledge. - how much was read: most of it read - source; notes: Agora: Toward Autonomous Bug Detection in Production-Level Consensus P; notes: Agora: Toward Autonomous Bug Detection in Production-Level Consensus P - record id: `agora-2026` - AlgoVeri (2026) - title: AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms - what: A benchmark of 77 classical algorithms. Experts wrote matching specs in Dafny, Verus and Lean. Models generate code and proofs, then repair them using verifier error messages. An LLM judge filters out cheating and degenerate algorithms. - number: 77 algorithms - number: Gemini-3 Flash 1x15: acceptance/filtered Dafny 55.84/40.26, Verus 25.97/24.68, Lean 9.09/7.79 percent - number: GPT-5.3 Codex 1x8: Dafny 49.35/42.86, Verus 14.29/11.69, Lean 23.38/11.69 - number: Best (Gemini-3 Flash, 15 repair rounds): 40.3% Dafny, 24.7% Verus, 7.8% Lean - limits: The specs are given, not discovered. The filter is a judgment call. Only termination of sequential programs is covered. Repair budgets differ between rows. No expert effort is reported. - how much was read: most of it read - source; notes: AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classic - record id: `algoveri-2026` - AlphaProof Nexus (2026) - title: Advancing Mathematics Research with AI-Driven Formal Proof Search - what: LLM agents write Lean proofs (a proof assistant whose compiler checks every step) for open math problems. The simplest agent, an LLM looping with Lean compiler error messages, matched a complex one with evolutionary search and a trained prover on the Erdős problems it solved, but cost more on the hardest. - number: 9 of 353 open Erdős problems solved by the full-featured agent, at a few hundred USD per problem - number: 44 of 492 OEIS conjectures proved - number: Basic agent (LLM plus Lean feedback) solved all 9 Erdős problems; full agent 2x to 5x cheaper on #138 and #125, roughly half as cost-efficient on the rest - number: AlphaProof alone, and basic agent on smaller Gemini models: 0 of 9; Codex (GPT-5.5): 7 of 9; Claude Code (Opus 4.7): 0 of 9 - limits: Math, not software or distributed systems. Only 353 problems that volunteers had already formalized in Lean; success concentrated where Lean's math library is mature. Statement faithfulness checked by hand by experts after each solve. Cost counts exclude the compute spent screening all 353 problems. Commercial-agent comparison is one setting each ('goal' mode). - how much was read: most of it read - source; notes: AlphaProof Nexus on open math problems (2026) - record id: `alphaproof-nexus-2026` - APE-Bench repository-scale proof engineering (2026) - title: APE-Bench: Evaluating Automated Proof Engineering for Formal Math Libraries - what: Defines Automated Proof Engineering (APE) for repository-scale Lean/Mathlib maintenance and releases APE-Bench plus APE-Harness. Contracts pin repository/toolchain, objectives, access boundaries and verification protocols. APE-Bench mines real Mathlib commits, synthesizes natural-language instructions and validates patches with dual checks: Lean compilation and LLM-judge semantic requirement/scope validation. APE-Harness provides isolated workspaces, version-scoped retrieval, execution services and interchangeable APE-Agent, Claude Code and Codex scaffolds. The benchmark contains 100 tasks from 67 Mathlib commits (2026-01-06–12), 10–284 changed lines, 17 mathematical domains, and is evaluated against miniCTX (334 tasks) and miniF2F (244). Under 3 and 100 turns per task, Gemini 3 Flash reaches 47% APE-Bench success, Gemini 3 Pro 24%, GPT-5.2 27%; miniCTX reaches 87.1%, 80.5%, 71.9%. Static gold-solution analysis finds 95/100 tasks require declarations from other files (median nine declarations across eight files). Difficulty declines with diff size and cross-file sources. Three judges achieve 90.6% overall semantic accuracy (92.2% correctness, 93.8% requirement alignment, 89.1% scope). APE-Agent outperforms Claude/Codex by 15–23 points; content-addressable storage cuts 480 GB to 71 GB and retrieval annotation cost $12.9Kto$197. Limitations are Lean/Mathlib-only, single-file modifications and LLM-judge scope errors. Appendix A traces a sqrt continuity/filter task solved in 32 turns and $0.79,whileAppendixBgivesjudgesacceptingunauthorizeddeclarationdeletionsandrejectingbenignrefactoring;thesecasesexplainwhycompilationandLLMconsensusstillneedcalibratedscopechecks.-number:100tasks,67commits,17domains;10–284difflines(median30)-number:APE-Benchsuccess:Flash47$3
  • number: miniCTX: Flash 87.1%, Pro 80.5%, GPT-5.2 71.9%
  • number: 95/100 gold solutions use cross-file declarations
  • limits: Evaluation pins Lean/Mathlib versions and blocks target files while exposing read-only context; agents have no internet and only APE-Harness services. Semantic correctness depends on majority-voted LLM judges, despite expert validation. Tasks are single-file modifications mined from commits, so multi-file engineering and other proof ecosystems are not covered. Fixed 3/100-turn budgets confound reasoning capability with cost efficiency; the paper itself finds model ranking is budget-sensitive. - how much was read: most of it read - - record id: `ape-bench-repository-scale-proof-engineering-2026` - Autogram (2026) - title: Invariant Discovery for Networked Systems - what: An LLM proposes typed expressions using telemetry metadata, not raw values. Then deterministic enumeration, logic checks and statistical tests keep the approximate relations that usually hold between live network measurements. - number: Full recall on Abilene and GÉANT vs NetNomos 19.8% and 15.1% - number: GÉANT: 101 relations vs NetNomos 1,645 (946 vacuous, 27 match reference) - number: Production WAN (about 100 routers, 1000 links): 41 relations in ten minutes; link hold rate 96.5% - number: full recall on Abilene and GÉANT vs NetNomos 19.8% and 15.1% - limits: The relations are approximate. They are not inductive invariants (facts that hold initially and are preserved by every step). The paper states no safety or liveness contract, program proof or measure of human effort. It admits false acceptances but gives no rate. Proofs are left to future work. - how much was read: most of it read - source; notes: Invariant Discovery for Networked Systems; notes: Invariant Discovery for Networked Systems - record id: `autogram-2026` - AutoSpec (protocol) (2026) - title: Synthesizing Precise Protocol Specs from Natural Language for Effective Test Generation - what: It extracts states, messages and constraints from RFCs into executable grammars. It tests Dovecot, Postfix, vsFTPd and Pigeonhole with Fandango, and repairs the grammars from traces. Experts wrote golden grammars to serve as the oracle (the reference answer). - number: POP3 295 lines/3.5 h; SMTP 373/9.5 h; IMAP 676/14 h; FTP 576/5.5 h; ManageSieve 449/5 h (golden grammars) - number: average recall 92.8% and 80.2% for client/server messages and independent constraints, 16.7% and 0.0% for dependent constraints (the notes list four values after 'Average') - number: abstract average 81.5% message acceptance - number: Message types recovered 92.8% (client), 80.2% (server) - limits: It misses dependent constraints entirely. Repair guided by execution may end up matching a buggy server. FTP data channels are excluded. Counts in the tables are inconsistent. The tool release is only planned. - how much was read: most of it read - source; notes: Synthesizing Precise Protocol Specs from Natural Language for Effectiv - record id: `autospec-protocol-2026` - Beyond Benchmarks / Vostd FVT (2026) - title: Beyond Benchmarks: A Case Study of LLM-Generated Verus Specification Failures on Asterinas Vostd - what: The authors ran three LLMs to write Verus specifications for six real verification targets in Asterinas, a Rust OS kernel, directly inside its repository, then hand-labelled every failed attempt by where it broke: the code did not fit the repository, broke Verus's own rules, modeled the wrong thing, or lacked proof structure. They also checked whether the verifier's error message pointed at the real cause. - number: 6 targets, 156 function-level tasks, 50 Rust files, 7,149 lines - number: 1,587 reviewed failure episodes: Stage I repository integration 442 (28%), Stage II Verus legality 502 (32%), Stage III semantic modeling 420 (26%), Stage IV proof construction 223 (14%) - number: benchmark verify rates 66.0% to 82.6% (Table 3 subtotals); on the 6 targets AutoVerus 22.7-24.5%, VeruSAGE 34.4-41.7%, the authors' workflow 66.3-68.7% - number: visible verifier diagnostic matched the true source stage in 1,152 of 1,587 episodes (72.6%) - limits: One project (Vostd, Entity-Model proof style) and three LLMs; stage labels come from the authors' manual review with no second reviewer reported; the authors' own workflow was built for this study. Covers specification generation, not distributed protocols or liveness, and the paper itself says the three LLMs are not a leaderboard. - how much was read: most of it read - source; notes: Beyond Benchmarks, Verus specification failures on Asterinas Vostd (20 - record id: `beyond-benchmarks-vostd-fvt-2026` - Can Code Specify a System Precisely Enough to Formally Verify It? (2026) - what: Audits a deployed restaurant point-of-sale payment workflow (register, payment terminal, Finix backend). The authors built TLA+ models with Claude agents under their supervision. They also compared LLMs generating specifications from the TypeScript source in three forms: library-based JavaScript, a bare JavaScript next-function, and TLA+. - number: pre-fix models have 43-103 states, check in under a second, counterexamples 6-9 steps - number: 7 original gaps found, 3 reproduced on the unmodified code with its emulator - number: final failure-enabled model has 376 distinct states; 24-hunk patch; passes all 23 scenario tests - number: 105 generations (7 models x 3 contracts x 5), each replayed on 101 observed transition windows - limits: Every checked property is a safety invariant (a fact true in every reachable state) on finite, simplified models, not on all executions of the TypeScript. Timeout liveness is not checked. The model and the emulator shared a wrong reading of how Finix responds to a duplicate key. There is no mechanical proof that code matches model. Crashes and power loss remain accepted risks. - how much was read: most of it read - source; notes: Can Code Specify a System Precisely Enough to Formally Verify It?; notes: Can Code Specify a System Precisely Enough to Formally Verify It? - record id: `can-code-specify-a-2026` - Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought (2026) - what: Introduces VCoT-Lift, which transforms low-level Z3 proof traces into explicit Verus-level proof steps, and VCoT-Bench, 1,988 completion tasks with semantic proof holes. It evaluates whether LLMs reconstruct missing invariant, assertion and lemma reasoning, rather than merely causing a verifier to pass. The four-stage pipeline uses a global proof transformer, rule-specific checker agents for high-level Z3 rules, a pruner for trivial/redundant steps, and a repair agent that compiles/checks transformed Verus proofs. A 36-rule Z3 hierarchy groups high/medium/low importance. Checker mappings must explicitly associate Z3 steps with Verus steps; a transformer-checker loop keeps global context while applying partial checks. VCoT-Bench derives 1,159 ratio tasks, 439 proof-type tasks and 390 location tasks from 150 verified Verus programs. Overall accuracy weights semantic and syntactic correctness (1, 0.5, 0.25, 0 for levels 3 through 0). GPT-5 mini is selected as semantic judge after 100-task, eight-judge validation. - number: 1,988 tasks: 1,159 ratio, 439 type, 390 location - number: 150 source programs; lifted corpus averages 6.5× proof lines, up to 32× - number: best greedy overall: Claude Sonnet 4.5 68.71% invariants, 39.55% assertions, 51.54% lemmas - number: at 100% block removal Claude Sonnet 4.5 17.22%, Qwen 3 think 0.66% - limits: The benchmark relies on Z3 proof traces, Verus syntax and a specialized GPT-5-mini semantic judge, so semantic accuracy is not itself a verifier theorem. The checker assesses only selected high-level rules; low-level steps are abstracted/pruned as trivial or redundant. Models are zero-shot and outputs are judged on completion tasks derived from existing verified programs, not independently generated distributed protocols. Context removal can overrepresent small one-block programs at 100%. The study reports current-model fragility and does not show that VCoT-Lift guarantees complete semantic equivalence for arbitrary proofs. Overall Acc gives partial credit (0.5 for semantically correct but syntactically invalid; 0.25 for syntactically valid but semantically incorrect); it is not the fraction of fully correct verified completions. - how much was read: most of it read - - record id: `can-llms-reason-like-2026` - Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation (2026) - what: # Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation (authored by agents unless marked 🧑) Bisharat, Ortiz, Spencer, Bhadauria, Wang, Thiruvathukal, Läufer and Abuhamad evaluate generation and completion of TLA+ specifications. The useful contribution is a public task, four prompt designs, checker outcomes, reference-text comparisons and a concrete error taxonomy. The experiment reports 2,600 open-weight runs and 130 proprietary-model runs. Its headline “semantic correctness” means an operational TLC outcome for a finite configuration. It does not establish correspondence to the natural-language requirements or equivalence to a reference behavior. Infrastructure failures, inconsistent counts and released evaluator differences materially limit the proposed ranking of strategies. The saved arXiv version is 2606.05792v1, dated 4 June 2026. The publisher first page identifies ICSOFT 2026, pages 39–50, DOI 10.5220/0015070400004088 (B1). This record reads the entire 12-page arXiv paper, including all 34 references and 11 tables. All 12 pages were inspected as rendered images. There are no figures, numbered theorems, mathematical proofs, appendix, checklist or separate algorithm listing. The publisher PDF was acquired and its first-page bibliography inspected; no byte identity or complete final-version comparison is claimed. The FormaLLM source is pinned to b159f5df093e7ed71f4793bba99459b97a2bb23d, dated 2 April 2026. Twenty-four selected plaintext files were read: documentation, split and prompt code, SANY/TLC evaluators and collectors, completion metrics, four pipeline files and three graph helpers. Identical files were checked by bytes and read once. The complete 1,864-entry tree was inventoried, not wholly read. Original/generated specifications, recorded checker outputs, full corpus/splits, run.sh, unselected analysis scripts and cited papers were not read. No author code was executed or imported; no models, checkpoints or serialized datasets were loaded. Source correspondence to the published historical runs remains unknown. quotes.json preserves 43 exact excerpts with source, line, character and section pointers; numeric_tables.json preserves all 11 literal table blocks and reader arithmetic. ## Task, inputs and experiment The corpus contains 205 specifications from 98 TLA+ Examples projects, including consensus protocols, concurrency examples and puzzles. Table 1 lists 92 with both comments and configuration, 14 with only configuration, 95 with only comments and four with neither; totals are 106 with configurations and 187 with comments. Thus “205 paired specifications” needs qualification. The paper reports a project-stratified 143/31/31 train/validation/test split, with five test exclusions leaving 26 evaluated specifications (Q1,T1). Training supplies examples; validation tunes prompts/hyperparameters. These are public repository comments rather than new, independently elicited requirements. No contamination or memorization analysis is supplied. Twenty-five open-weight models use four strategies; five proprietary models use only few-shot prompting. This gives 25×26×4=2,600 plus 5×26=130 runs. Table 2 preserves model aliases, reported token counts and contexts. Its many `latest` aliases, missing Qwen3:4b context cell and unexplained extreme totals such as 1,141.7K for r1:1.5b limit exact model/budget reconstruction. These token summaries are not measured end-to-end time, price or energy. The “eight families” combine architecture families with overlapping categories such as code specialization and proprietary APIs; they are the authors' grouping, not eight independent experimental factors. Few-shot (FS) supplies three training comment/specification examples and target comments. Progressive (Prog.) supplies three examples plus instructional messages about module structure, variables, operators and temporal properties. Fill-in-middle (FIM) supplies the first and last 30% of the reference and asks for the middle 40%. Half-completion (HC) supplies the first 50%, comments and configuration and asks for the remainder (Q2–Q4). Completion therefore receives substantial correct reference code and tests a different task from full generation. In selected source, full-generation prompts also include the original configuration, while the written generated configuration becomes the preferred checker input (C10–C11). The paper says SANY passes on “Parsing completed” with a 30-second timeout, then sends passing specifications with configurations to TLC (Q5). Textual measures apply to the generated reference segment: BLEU, ROUGE-L, edit distance, exact match and line accuracy. Source C13 defines line accuracy by matching line subsequences divided by reference line count, character similarity by SequenceMatcher, character edit distance, whitespace-token BLEU and default ROUGE tokenization. These are textual measures; different but equivalent specifications can score poorly, and reference overlap does not establish correct behavior. ## What the tables report Table 3 reports SANY FS173/650, HC80/650, Prog162/650 and FIM92/650. TLC is FS0, HC0, Prog56 and FIM0, with the Prog rate 8.6%. These are printed experimental outcomes, not independently rerun results. Table 5 distributes those outcomes across models, and Table 10 across specifications. The strongest listed individual open-model TLC result is DeepSeek r1:8b with14/26=53.8% under Prog; Qwen235b has12, Qwen30b10, GPT-OSS20b6 and Phi-mini4. These individual rates differ from the aggregate 8.6% headline. Table 6 is essential to the headline's scope: under FS, GPT-5 reports SANY26/26 and TLC7/26; Sonnet4.5 reports24/26 and3/26, Haiku4.5 reports21/26 and3/26, Opus4.1 reports23/26 and1/26, GPT-4o reports20/26 and1/26. Thus “only progressive” applies to the open-weight multi-strategy experiment; it is false as an unqualified statement about all30 models. Likewise 26.6% is an aggregate open-strategy syntax rate, not the maximum individual result. The sentence calling GPT-5's26.9% TLC “comparable to the open-weight FS ceiling” conflicts with the reported open FS0% (Q7,T3,T6). Tables 7 and9 preserve all per-model completion similarities. Table 3 reports mean BLEU HC.062/FIM.077, ROUGE-L .189/.213, line accuracy10.90%/12.27%, and exact matches0/650 versus1/650. Table 7 instead prints FIM average12.37%; reported dispersions also differ across summary and detailed tables. The one FIM exact match together with zero TLC passes is another reason to inspect infrastructure/configuration before interpreting zero as behavioral failure. Table 8 reports bad-module-body first errors FS228,HC264,Prog243,FIM290; other parse errors72/227/70/191; unexpected-token errors82/29/65/17. These are not all categories. Its first-error location counts sum to477/570/488/558, matching 650 minus the Table 3 SANY totals. A first detected error does not count all defects. Early/middle/late thirds are the paper's definition; the selected collector instead labels absolute line numbers below10 and below50 (C14), so the published calculation needs a different, identified analysis path. Table 10 gives examples rather than a simple “distributed is harder” result. YoYoNoPruning has100% SANY for HC/FIM; HDiskSynod has FS32%, Prog20% and FIM48%, contradicting the prose example that it scores0%. BinarySearch has12%FS/24%Prog syntax and8%Prog TLC. The highest listed per-spec Prog TLC rate is16%, shared by EnvironmentController, MultiPaxos, VoucherCancel and YoYoNoPruning. MCDieHarder has zero TLC across strategies. Multiple spelling/wrapping forms are retained in literal rows; no semantic equivalence across similarly named modules is assumed (Q10,T10). ## Infrastructure confounding and inconsistent totals Table 3's footnote explicitly says zero TLC rates “reflect failures (i.e., State Directory Conflicts and Module Not Found errors).” Table 4 labels97.2%/99.2%/99.2% of FS/HC/FIM failures as infrastructure and9.8% for Prog. A pipeline unable to load a module has not tested its behavior. Calling invalid configuration a “genuine semantic” failure also does not establish a mismatch between the generated transition system and intended requirements. These failures can be important practical generation defects while remaining distinct from behavioral incorrectness (Q6,T3–T4). Table 4 is internally incomplete or inconsistent. Its displayed module/configuration rows sum FS177,HC164,FIM161 against totals650 each; an unlisted state-directory category might explain some gaps but is not numerically supplied. For Prog it gives122 TLC failures,455 parse failures and56 passes, totaling633 rather than650. Table 3's162 SANY passes imply488 parse failures. Moreover122+56=178 exceeds162. The record retains these printed claims and cannot reconstruct a valid common denominator from them. Reader arithmetic over Table 5 gives column totals FS173/0,HC78/0,Prog162/56,FIM92/0. Table 10 implies HC80, matching Table 3 but conflicting with Table 5. Table 5's averages6.92,3.17,6.38,2.08,3.71 differ from row-derived6.92,3.12,6.48,2.24,3.68 for the corresponding nonzero columns. Also80/650=12.31%, whereas Table 3 prints12.2%; Qwen30b's23/26 is88.46%, whereas its best-model cell prints92.3%. No source output logs were inspected to settle which transcription or analysis is wrong. ## Claims about models, causes and error repair The observed results support some nonmonotonic size comparisons. They do not support “r1:8b outperforms70b across all strategies”: HC SANY is1 versus3 and FIM3 versus3. The claim that LLaMA1b's12 Prog SANY passes are highest among all LLaMAs conflicts with both3.3 variants'18. “Just three model families” conflicts with the paragraph listing DeepSeek,Qwen,GPT-OSS,Phi and Table 5's Gemma pass. Model training, instruction alignment, context limits, prompt interaction and output processing are not separated by a controlled factorial study (Q6,Q8–Q9,T5). The authors attribute poor code-model performance to negative transfer, operator substitutions to rendered mathematics and reasoning leakage to training. These are proposed explanations. They did not manipulate training data, measure causal mediation, run controlled fine-tuning ablations or establish statistical significance. “Not noise” is especially strong given their explicit limitation: “We run each model-specification pair once per strategy” (Q14). More heterogeneous tasks do not substitute for repeated measurements of a given model/task condition. Table 11 and §4.4 give five categories: Unicode operator substitutions, syntax from other languages, reasoning/formatting leakage, length miscalibration and module structure. Semicolon first-error counts are18/11/25/10, while file-presence scans count49/23/82/22; these have different denominators and meanings. Backtick counts are28/10/21/13; think-block counts15/30/not-applicable/28. The paper says QwQ contributes26/26 HC and FIM think blocks. It reports mean output/target length HC3.46× and FIM2.24×, with9.84× as an extreme. These are observed author counts, not a controlled explanation of accuracy (Q11–Q13,T11). The claimed243+ missing Prog terminators numerically matches Table 8's243 bad-module-body first errors, but that broad parser category is not equivalent to an independently counted missing terminator. The selected generator also appends a terminator. The historical counting path is unresolved. A semicolon-presence scan is not itself a syntax-error test: semicolons can occur within comments, strings or PlusCal text. Similarly a blanket claim about all Unicode needs a tool-version/context boundary. Suggested grammar constraints, retrieval, normalization, fence removal and checker-guided repair are future interventions here. Their correctness and benefit were not measured; indiscriminate character deletion could change intended content (Q13,Q15,C16). ## Released source qualifications The primary parse step checks return code0 plus a success substring (C1–C2). Its separate metrics collector computes `parse_success` from that substring and error keywords but never uses it to return status: it returns PASSED whenever its line-number regexes find no errors (C3–C4). By static inspection, a present empty/unrecognized-error log can therefore receive PASSED in that collector. This is a source-level possibility; it is not a measured false-positive rate in the paper. The selected TLC loops iterate every parsed dictionary entry and ignore `parse_status`, contrary to the paper's stated filtering. They prefer any generated configuration file, even an empty one, before fallback to the corpus; full generation writes one. Hence the checker may assess model-selected properties/configuration rather than an independently fixed behavioral oracle. The success collector accepts “Model checking completed. No error has been found.” before checking exit status. The alternate progressive evaluator instead accepts “The specification is correct”; the two strings are different. TLC subprocess calls have no Python timeout, despite TIMEOUT classification later; clearing `generated/states` without setting a working directory/metadir does not establish isolation of TLC's actual state directory. These observations require run-path/log validation before assigning them to historical results (C5–C7,C11). The split helper performs unseeded flat `random.shuffle`, not project stratification, and floor slicing205 gives143/30/32 rather than143/31/31. Selected prompts read val.json rather than test.json. Those labels may have been repurposed by uninspected run tooling; this record cannot certify the published test set from the selected code (C8,C15). The `fifty-fifty` and `fill-in-middle` prompt files are byte-identical and implement30/40/30 completion; their metric files are also identical. This does not reproduce two distinct completion designs. The progressive prompt accumulates instructional messages and makes one `llm.invoke` per target; it does not execute separate checked module/variable/operator generation stages. Its cleaner removes fences and explanatory text, adjusts headers and appends `====`, as do other selected generators. Thus Table 11's FIM cleanup footnote does not establish cleanup unique to FIM. The source/documentation mix should not be silently treated as the exact experimental pipeline (C9,C11–C12,C16). ## Contribution to this literature effort The paper is directly relevant to evaluating LLM-written distributed-system specifications. It supplies task definitions, public examples, prompt contrasts and failure cases. Its strongest lesson is methodological: separate parser success, tool execution, checked finite properties and faithfulness to intended behavior. The evidence does not justify treating an8.6% pipeline pass fraction as a measured semantic-equivalence rate, or infrastructure-confounded zero passes as proof that a strategy cannot generate valid behavior. A reader recommendation is to use fixed independent configurations/properties, inspect explored states and vacuity, compare behaviors or traces against requirements, isolate checker paths, publish per-run logs and exact model versions, repeat runs, and report infrastructure failures separately. The recommendation extends beyond the paper's measured interventions. This study verifies no Verus/Rust implementation and proves no distributed protocol. It evaluates draft TLA+ artifacts rather than an autonomous multi-agent proof process. Prose consultation is complete; independent review is pending. REVIEW.md records consultation scope and disposition. This is a whole-paper reading with selected static source inspection, not an experimental reproduction. Fresh independent review found no material conclusion issue; two table-packaging defects were corrected (REVIEW.md). All11tables/34refs,43literalquotes,16diagnostics;operational-vs-behavioral correctness,infra confounding,count arithmetic,proprietaryresults,source evaluator/split/prompt discrepancies. - number: 143 training, 31 validation, 31 test specs; 26 eligible test specs used - number: GPT-5 26/26 syntax and 7/26 TLC passes (26.9%) - number: 8.6% aggregate open-model TLC success (56/650); DeepSeek r1:8b 14/26 - number: 2,600 open-model runs plus 130 proprietary runs - limits: Public comments and26finite configured test specs;one run percondition;inconsistent counts;infra failures;NLfaithfulness notestablished;no authorcode/checker rerun;selected release correspondence uncertain. - how much was read: most of it read - source; notes: Natural language to temporal logic and TLA+ - record id: `can-llms-write-correct-2026` - CodeSpecBench (2026) - title: CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation - what: CodeSpecBench is a benchmark where a model writes executable Python pre- and postconditions. The input is either a function description or a real repository issue. A harness checks that the conditions accept valid tests and reject invalid ones. - number: 2,494 function tasks and 500 repository tasks from 12 projects - number: 96.3% statement and 93.6% branch coverage - number: 67.8 incorrect test cases per problem - number: best strict pass: 47.0% (GPT-5-mini) on function tasks, 20.2% (Claude-4.5-Sonnet) on repository; Appendix table says 21.0 - limits: It uses finite tests only. Localization and tests are supplied. The repository figures are inconsistent across tables. The comparison of solution generation mixes in numbers from model reports. - how much was read: most of it read - source; notes: CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specificati - record id: `codespecbench-2026` - CrossCheck (2026) - title: CrossCheck: Input Validation for WAN Control Systems - what: CrossCheck compares the inputs of a WAN controller with router telemetry that was collected independently. It repairs bad measurements by voting. It checks demand and topology consistency against relations that humans chose. - number: four-week shadow deployment, zero false positives, one demand double-counting incident - number: repair about 9.1 s, validation 100 ms, queries 56 ms - number: detects 100% of demand perturbations of 5%+; tolerates up to 30% missing counters - number: about 250 repair and 100 validation Python lines - limits: It is an operational check, not a proof. It is not in the live decision path. Humans define the relations and tune the thresholds. It uses no LLM. - how much was read: most of it read - source; notes: CrossCheck: Input Validation for WAN Control Systems; notes: CrossCheck: Input Validation for WAN Control Systems - record id: `crosscheck-2026` - DafnyBench (2026) - title: DafnyBench repository - what: DafnyBench provides 782 Dafny programs with verified ground-truth and hints-removed versions; evaluation asks models to restore annotations and runs Dafny. - number: 782 Dafny programs - limits: Annotation completion is a narrower task than intent-to-spec and has no distributed liveness by itself. - how much was read: most of it read - source - record id: `dafnybench-2026` - Deductive Verification of Temporal Collisions in Distributed Message Brokers (2026) - what: The authors hand-write TLA+ templates for Kafka, RabbitMQ, NATS JetStream and Pulsar consumers. They use TLC to check for duplicate execution after a crash that happens before acknowledgment. Six Docker experiments cross-check the results on real consumers. - number: 112 configurations, 16 models, states 16-211,201 - number: 120-second timeout - number: Table 1: faulty 10/10 violated; idempotent 3 violated/7 pass; dual-async 7/3; synchronization 55/27 - number: 6/6 physical predictions matched - limits: Despite the title, this is bounded model checking. Liveness and ordering are not checked. The RabbitMQ idempotent models have three false positives. The paper claims that 30 PASS results fail at higher bounds, but does not check this. The wording of Table 7 conflicts with its results. - how much was read: most of it read - source; notes: Deductive Verification of Temporal Collisions in Distributed Message B - record id: `deductive-verification-of-temporal-2026` - DissProve (2026) - title: DissProve: Automated Verification of Asynchronous Distributed Protocols with Affine Communication - what: DissProve performs backward abstract interpretation of P actor protocols from a negated safety property. Separation logic tracks local actor state/control, linear logic tracks pending messages, and ordered histories constrain causal execution. Goal-directed materialization, causal reduction, message segments and entailment refute unsafe states for unbounded actor populations. [Q1,Q3–Q18,Q23–Q25] Complete election quorum-bug/refutation, aliasing-aware actor materialization, affine sender-count bounds, handler-independence requirement for message segments, five causal graph edges, counter-summary versus entailment cycles, conditional acyclic-graph termination, full appendices and exact mechanization assumptions. All seven evaluation rows, query counts and capped ablations are preserved. Corrects unsupported Raft-timeout and Theorem 4.1-only mechanization claims. [Q3–Q49] - number: 7 verified P protocols,49–99 SLOC - number: 16 states Bakery;9275 Atomic Broadcast - number: Two-Phase Commit 5888 states,1:08:46;Atomic Broadcast 9275 states,1:12:46 - number: 25000 state cap without entailment;Barrier 87 versus 35 states - limits: Affine means at most one message type per sender/receiver in the modeled round; initialization precedes other messages, Main-only creation/deployment is outside analysis, no shared memory or message duplication. Properties are prenex universal; no liveness proof. Segment soundness requires independent/reorderable handler invocations and sound recurrence abstraction. Termination requires acyclic causal dependencies and terminating handler analysis. Authors report Lean proof of Theorem 4.1 with assumed concretization lemmas and also Worklist-LemmaB.8; Section 5 segment semantics and full termination are not fully mechanized. Paxos several-hour noncompletion is reported; no separate Raft run is stated. Formula notation and graph/branching definitions have reader-identified ambiguities, without any artifact conclusion. [Q1,Q3,Q9–Q10,Q16–Q22,Q32–Q47] The introduction defines affine communication per sender/type/receiver [Q1], but §3.1.3 says no two messages of the same event type may carry different payloads from one sender, without the same-receiver qualifier [Q10]. The latter wording additionally excludes receiver-specific different payloads. This apparent prose discrepancy remains unresolved; no implementation condition is inferred. - how much was read: most of it read - source - record id: `dissprove-2026` - DRPBench: Evaluating LLMs in Concurrent Code Comprehension via Fine-Grained Data Race Prediction (2026) - what: Introduces DRPBench, a static fine-grained data-race prediction benchmark for concurrent-code comprehension: 1,003 SV-Benchmarks C/C++ programs, 226 racing and 777 race-free, with 557 manually annotated variable/line-level races. Evaluates six LLM families in standard/reasoning/agentic configurations (15 LLM configurations) on a balanced 200-program DRPBench-Lite subset, plus Deagle and UGemCutter. Gemini-3-thinking reaches F1 74.89%; Llama 3 70B reaches 8.80%. Defines a race as a variable and exact conflicting access-line pair, parses JSON predictions as sets, and reports precision/recall/F1. Annotation combines three human annotators, o3-mini, GPT-4o and Deagle, two senior meta-reviewers, and consensus. DRPBench-Lite stratifies 100 racing and 100 race-free programs. Prompts provide definitions of mutexes, semaphores, atomics, condition variables and thread lifecycle. Ablations compare domain-knowledge prompts; tables compare reasoning and OpenHands agentic scaffolds against standard models. Qualitative cases identify shared-variable distraction and synchronization-logic myopia. - limits: The corpus is isolated pthread-based C/C++ and assumes sequential consistency; races only under weak memory models are excluded. Ground truth is an expert/LLM-assisted annotation process, not a machine-checked universal oracle. The reported LLM evaluation and error analysis use DRPBench-Lite (100 unique racing plus 100 race-free programs), not all 1,003 programs. The full corpus statistics and 557 race annotations should not be read as the test-set size. A true positive requires exact shared-variable name and exact pair of conflicting line numbers; this strict score measures localization under the supplied race definition, not general concurrent-code correctness, bug severity, or absence-of-race proof. Six base LLMs yield 15 standard/reasoning/agentic configurations; Deagle and UGemCutter are separate symbolic baselines. OpenHands agentic runs are only for Gemini, Claude and DeepSeek, with default decoding/agent configurations; other models lack agentic results. Annotation uses LLM tools (o3-mini and GPT-4o) alongside humans and Deagle, then two expert meta-reviewers; source SV-COMP labels and the paper's manual review are not independently reproduced by reader. The benchmark authors report >500 person-hours. No quantitative uncertainty bars or statistical tests are reported for the main F1 comparisons; no external artifact execution or independent annotation was performed by reader. LLM outputs remain candidates and cannot prove race absence. Appendix says this version covers isolated pthread C/C++ and SC only; real multi-component software, Java/Go, and weak-memory-only races are future scope. Symbolic-tool limitations and timeout/heuristic effects complicate direct superiority comparisons. - how much was read: most of it read - - record id: `drpbench-evaluating-llms-in-2026` - EqiVM (2026) - title: Foundational Refinement Proofs for Deployed Bytecode, at the Price of Tokens (EqiVM) - what: Agents write Lean proofs that deployed Ethereum contract bytecode refines (correctly implements) a spec written in the authors' Sol- language. The proofs use a formal semantics of the EVM. The agent is an untrusted proof producer, so its proofs are checked. - number: text says 19 contracts proved complete, abstract says 23, conclusion says 27 (notes' own count: 27) - number: Vat: 6,965 bytecode bytes, 556 spec lines, 84,382 Lean proof lines, 100h44m, 2 effective human prompts, 74.8M fresh tokens, about 1,380 retrospective API price
  • number: Cat: 106.8M tokens, 64 h, 17 prompts
  • number: Clipper 26/29 functions after 81h44m; Nouns Auction 18/20 after 94h28m with certificate check timing out
  • limits: It does not show transaction availability, bounded gas, eventual completion or soundness of the supplied policy. The refinement relation permits vacuity through gas exhaustion (running out of gas makes a claim hold trivially). The trusted base includes the EVM formalization, imported Lean axioms and native_decide. A human supplies the high-level spec.
  • how much was read: most of it read
  • source; notes: Foundational Refinement Proofs for Deployed Bytecode, at the Price of ; notes: Foundational Refinement Proofs for Deployed Bytecode, at the Price of
  • record id: eqivm-2026
• Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving (2026)
  • what: The paper audits Lean benchmarks using Lean metaprograms (a method called semantic guard proving) and LLM screening. It finds defective statements, then corrects some of them.
  • number: Table 4: 10,318 problem occurrences, 4,833 findings, 398 mechanically certified (prose says 399)
  • number: semantic screening 92 problems, 552 classifications; Sonnet precision/recall/F1 0.30/0.82/0.42, GPT 0.24/0.91/0.37 (Appendix H gives Sonnet recall 0.87)
  • number: correcting 20 unprovable statements: 0 to 3/20 and 2/20 for two provers
  • number: 4,833 findings, including 398 mechanically certified issues
  • limits: The benchmarks are mathematical, not software. Duplicates across forks mean the counts are not the number of unique errors.
  • how much was read: most of it read
  • source; notes: Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Fail
  • record id: faults-in-our-formal-2026
• FM-Agent (2026)
  • title: FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning
  • what: An agent writes function contracts in natural language from callers, code and domain knowledge. It reasons about blocks and loops in Hoare style (what holds before and after each piece of code). It then generates and runs system-level tests to confirm suspected bugs.
  • number: 277k source lines and 8,549 functions across four systems; 522 new bugs total
  • number: Claude’s C Compiler 339 bugs (143k lines), VibeTensor 141, VibeOS 23, Bespoke OLAP 19
  • number: cost 2 days and 3.4 billion tokens total
  • number: ablation on compiler: 57 bugs vs 339, removing two components at once
  • limits: It handles sequential functions only. It cannot ensure soundness, and its reasoning is not machine-checked. The prose says 4,910 functions but the table says 4,957.
  • how much was read: most of it read
  • source; notes: FM-Agent: useful bug finding, without sound proofs; notes: FM-Agent: useful bug finding, without sound proofs
  • record id: fm-agent-2026
• Formal Verification of Multi-Controller Systems in Cloud Management (2026)
  • what: A NuXMV model of two Kubernetes-style controllers, two resources and two users. It checks convergence (eventually the requested state holds forever) and completion. Overlapping resources cause endless overwriting. Dependencies can block forever.
  • number: Table 2: four settings; no overlap/dependencies all pass, others fail
  • number: no runtime, effort or source size reported
  • limits: A small abstract model, built by hand from incidents. The unavailability result holds only if the model lets a resource be unavailable forever.
  • how much was read: most of it read
  • source; notes: Formal Verification of Multi-Controller Systems in Cloud Management
  • record id: formal-verification-of-multi-2026
• From Verification Failures to Reusable Guidance for Coding Agents (2026)
  • what: Turns diagnosed proof failures into reusable procedures (a kit) for coding agents. The agents write specifications and proofs of execution in the K framework, with Lean for leftover math. Separate agents audit the result.
  • number: 164/164 final AI audit PASS verdicts after two targeted repairs and heavy human-guided resource development
  • number: at the common checkpoint with the kit: 97 Pass, 67 Concerns, 0 Fail
  • number: 12-pair challenge: all 12 defects detected, all 12 clean accepted (Wilson interval 75.8-100%)
  • number: controlled 31-program test: GPT-5.6 Luna 15/31, 15/31, 19/31; all four intervals include zero
  • limits: The target is partial functional correctness (correct if it terminates), and termination is handled separately. The trusted base includes the translation, the semantics and the solver integration. The controlled evidence does not show a clear general benefit of the kit.
  • how much was read: most of it read
  • source; notes: From Verification Failures to Reusable Guidance for Coding Agents; notes: From Verification Failures to Reusable Guidance for Coding Agents
  • record id: from-verification-failures-to-2026
• GSIRS TLA-LLM poster (2026)
  • title: Large Language Models (LLM) and Temporal Logic of Actions (TLA): How Effective are LLMs for Verification Systems
  • what: This poster reports a systematic natural-language-to-TLA+ evaluation over 205 community specifications, 30 models and four prompting strategies. It validates syntax with SANY and behavior with TLC, separating parsability from semantic correctness and cataloguing recurring generation failures.
  • number: 205 TLA+ specifications across 98 projects
  • number: 30 LLMs from eight families; 2,600 open-model runs and 130 proprietary-model runs
  • number: up to 26.6% syntactic correctness but 8.6% semantic correctness; TLC passes occurred only with progressive prompting
  • number: semicolon injection occurred in 12.6% of generated files; backticks in 5.9%; 234+ missing module terminators were reported
  • limits: The poster is based on a separate published ICSOFT 2026 paper and gives an abstracted poster-level account; its evaluation uses finite TLC configurations and does not establish deployed-code correctness. The dataset is drawn from the TLA+ Examples repository, so its task mix is not a random sample of industrial requirements.
  • how much was read: most of it read
  • source
  • record id: gsirs-tla-llm-poster-2026
• HarnessLLM (2026)
  • title: HarnessLLM: Rust Verification Harness Generation with Large Language Models
  • what: An LLM workflow turns a Rust project’s existing unit tests into harnesses (small driver programs with symbolic inputs) for the Kani bounded model checker. It finds the call patterns in tests, builds symbolic arguments for the project’s own types by following constructors, and repairs compile errors without letting the LLM invent types.
  • number: 9 Rust crates, 494 tests, 6651 lines of test code: 294 calling scenarios extracted, 94.66% preserved the original behavior
  • number: 294 of 294 harnesses compile in Kani within 10 attempts; 97.3% within 3 attempts; about 1.4 generations per harness
  • number: Average 145 s and 0.03 per harness (GPT-4.1) - number: Kani Autoharness baseline produced harnesses for 41% of the same scenarios - limits: Needs a codebase with good tests; sparse tests lower scenario coverage. 'Success' means the harness compiles in Kani, not that it checks anything beyond memory safety and panics. Bounded checking; the LLM picks the bounds (slice lengths, unwind). Only functions visible in source (macro-generated ones are missed). - how much was read: most of it read - source; notes: HarnessLLM (2026) - record id: `harnessllm-2026` - IC3Syn (2026) - title: Synthesizing Inductive Invariants for Distributed Protocols via IC3 and Large Language Models - what: An IC3 controller hands an LLM (Claude Opus 4.6) concrete bad states to block. The LLM proposes TLA+ clauses. TLC and Apalache check them on small instances. TLAPS then proves the invariant for all sizes. - number: 29 of 29 protocols solved (Table 1 p. 9); direct Claude baseline 23/29; Gemini 3 Flash inside IC3 solves 27/29 - number: MLDR (MongoDB Raft model): 4525 s, 16 clauses, proof over 3,000 lines; EPR Paxos: 1806 s, 9 clauses - number: Default synthesis timeout 30 minutes, 2 hours for the two hardest cases - number: Paper's rerun of Endive solves 24 shared cases - limits: The models and safety goals are supplied. Liveness is not evaluated. The synthesis loop works on finite instances only. The proof stage used a scaffold plus Codex automation. The model, budget and log of human intervention are not stated. Full end-to-end automation is ongoing work. No public code or proof artifact was found. The results are the authors' own reports. - how much was read: most of it read - source; notes: IC3Syn: a direct 2026 LLM invariant search result; notes: IC3Syn: agent-written proofs of unbounded distributed protocol safety; notes: IC3Syn protocol inventory - record id: `ic3syn-2026` - InvariantEval (2026) - title: Smart Contract Invariants Protect Against Cybercriminals - what: Builds InvariantEval: 28 real Ethereum attacks, each paired with a one-line human-written check (an invariant) that would have blocked it. A replay tool (PonderReplay) re-runs the attack and 108,637 past transactions against the patched contract to confirm the check blocks the attack and keeps normal use working. Then tests three automatic invariant tools. - number: 28 attacks, about 1.07 billion USD total loss; all 28 blocked by the human invariant (PoC and real attack transaction) - number: 108,637 historical transactions replayed: 106,815 preserved (98.3%), 1,526 changed (1.4%), 296 inconclusive - number: inconclusive causes: 173 original out-of-gas, 12 subcall out-of-gas, 111 state drift - number: automatic tools recovered 2 of 28 ground-truth invariants (FLAMES only); InvCon 0, InvCon+ 0 - limits: Ground truth is hand-written by the authors after the attack, with the post-mortem in hand, and only for cases they could solve; the 28 are the ones that worked out of 359 Ethereum exploits. Benchmark and tool are available only on request. Matching an invariant to ground truth was done by checking whether it appears in tool output. - how much was read: most of it read - source; notes: Smart Contract Invariants Protect Against Cybercriminals (2026) - record id: `invarianteval-2026` - Ivy issue 82 (2026) - title: [ivy1.7, ivy1.8] unsoundness with two after init blocks - what: The issue gives a minimal Ivy program where a specification after-init assertion of false can be accepted when an implementation after-init action sets the related variable. - number: An unconditional false assertion is accepted in the supplied reproducer. - limits: Open issue; behavior is version-specific and the issue does not establish a released fix. - how much was read: most of it read - source - record id: `ivy-issue-82-2026` - JS-SAM (2026) - title: Executable JavaScript as a Checkable Specification Language: A JS-SAM Case Study on SysMoBench - what: This controlled study compares LLMs writing JavaScript (SAM) specs against TLA+ specs of real systems. The authors check the specs by bounded exploration, plus replay against traces captured from the running systems. - number: 4 models, 5 generations per cell - number: Spin traces: 28 windows, 8 of 18 observable input combinations covered - number: small tasks 40/40 complete passes; Raft scale 0/20 initially, 6/20 after one repair round - number: state-changing conformance of original models ranges 0-86.4% though all pass syntax and invariants - limits: It checks no liveness or temporal properties. It explores only to a bound. It does not prove that the implementation matches the model. The author designed the instrumentation, the reference model and the prompts. - how much was read: most of it read - source; notes: JS-SAM: a controlled case study separating specification shape from la; notes: JS-SAM: a controlled case study separating specification shape from la - record id: `js-sam-2026` - k8s-runtime-audit (2026) - title: Monitoring and Verification of Multitenant Kubernetes Clusters using TLA+ Trace Checking - what: A TLA+ model of a tenant-isolation policy is replayed over Kubernetes audit logs. The logs travel through a webhook and NATS JetStream to a customized TLC. It flags cross-tenant access, dangling bindings and cluster-admin bindings. - number: 5 controlled scenarios, all detected - number: one control-plane and two worker nodes; 10/50/100/1000 actions, 2/5/10 tenants - number: batch cap 50 messages, 4.5 s pull expiration - number: 2 tenants, 1000 actions: 1071.20 messages, 24.5 ± 0.8 ms per message; 10 tenants: 689.60, 70.3 ± 1.7 ms - limits: It checks recorded traces. It does not prove the implementation or the spec correct. It relies on logs that are complete and in order. The model omits namespace-local Roles. It has no liveness. No size or hours are given. - how much was read: most of it read - source; notes: Monitoring and Verification of Multitenant Kubernetes Clusters using T - record id: `k8s-runtime-audit-2026` - KaPilot (2026) - title: KaPilot: LLM-Assisted Generation of Kani Specifications for Unsafe Rust Verification - what: Five cooperating LLM agents write Kani contracts (preconditions, postconditions, loop invariants) for unsafe Rust functions, using the function's documentation instead of its code as the source of what must hold. Kani is a bounded model checker: it checks all inputs up to a size bound. - number: GoldSet (54 functions from verify-rust-std, with human-written contracts): 88.9% of functions got a contract Kani accepts; 57.4% of contracts equal to or stronger than the human ones (GPT-5: 31 good, 17 bad, 6 failure) - number: ULSet (70 functions from popular crates, no reference contracts): 71.4% got an accepted contract (GPT-5: 50 pass, 20 failure) - number: Versus AutoSpec adapted to Kani, same GoldSet and GPT-5: 31 vs 17 good, 6 vs 14 failures - number: Ablation on GPT-5: dropping SafetyReq cut good from 31 to 6; naive prompt with Kani feedback (10 rounds) gave 8 good, 26 failures - limits: Only memory safety and panics, not functional correctness. Kani is bounded, loop invariants only for while loops, and some Rust properties (lifetimes) cannot be written in Kani. Judging whether an accepted contract is good was done by hand; no automatic check exists. Output quality depends on documentation quality (ULSet docs shorter, failure 28.57% to 31.43% vs 7.41% to 20.37%). - how much was read: most of it read - source; notes: KaPilot (2026) - record id: `kapilot-2026` - KOSV (2026) - title: KOSV: Kubernetes Ownership Safety Verifier - what: It polls Kubernetes ownership state and normalizes it to JSONL traces. It checks two predicates written by the authors: at most one controller owns a resource, and no unexpected ownership transfer. Faults seeded into the observations test the checker itself. - number: 20/20 runs pass (5 each: baseline, 500 ms delay, 2,000 ms delay, controller-manager restart) - number: seeded faults 2/2 detected; denser-poll 6/6 pass - number: experiments pinned to v0.1.10 - limits: It is sampled, finite testing. PASS is not proof that nothing is wrong. Transients shorter than the sampling interval can be missed. Delay faults do not affect controller-API traffic. The reproduction table is empty. Two of the four oracles are unimplemented. - how much was read: most of it read - source; notes: KOSV: Kubernetes Ownership Safety Verifier - record id: `kosv-2026` - LeaseGuard (2026) - title: LeaseGuard: Raft Leases Done Right - what: LeaseGuard is a Raft lease design. A new leader can accept writes at once, and can serve reads that the old leader's writes do not affect, from the log it inherited. The authors built a TLA+ model, a Python simulator and a LogCabin C++ implementation. - number: TLA+ bounds: 3 servers, 2 keys, term 3, log length 4, clock 8, lease 3; 296-line model - number: about 1,000 simulator lines; 1,600 server and 650 client lines changed in 27,000-line C++ code - number: writes about 1000 to 10,000 per second - number: failover: 9,930 of 10,000 reads succeeded in the wait - limits: The safety argument is a paper sketch, not machine-checked. The TLA+ model checks safety only to a bound, with no temporal properties. There is no agent workflow. An earlier model hid the bug inside large atomic actions. - how much was read: most of it read - source; notes: Related-work addition: LeaseGuard: Raft Leases Done Right - record id: `leaseguard-2026` - LeetProof (2026) - title: Certified Program Synthesis with a Multi-modal Verifier - what: LeetProof checks generated specifications against examples and reference implementations. It then synthesizes Velvet/Lean code, using SMT solvers to discharge proof goals automatically. A person finishes the remaining goals interactively. - number: GPT-5.2 Velvet pipeline fully proves 28 of 50 tasks vs Lean-only 17 - number: 16 issues in VERINA (of 189 specs, 8.5%) and 18 in CLEVER (of 161, 11.2%) via property-based testing - number: Verina issues took 9 minutes of property-based testing vs about 8 hours of LLM equivalence checking - limits: It synthesizes algorithms with simple types. It has no distributed models and no liveness. - how much was read: most of it read - source; notes: Dafny, C verification, and the specification-intent gap - record id: `leetproof-2026` - LemmaNet (2026) - title: Automated Lemma Discovery in Agentic Program Verification - what: GPT-5.2 plans reusable helper lemmas from C source, ACSL obligations and Rocq proof states. It adapts them to each obligation and proves them first. - number: 298/641 SV-COMP and 66/300 NTP4VC tasks, 364/941 (38.7%, notes' arithmetic), 10-minute/100-step budget - number: 364 total proofs (298 SV-COMP, 66 NTP4VC) - number: 26.8% improvement over AutoRocq, 51.7% over Copra - number: 78 unique successes; 61/69 helper-lemma proofs use BRIDGE strategy - limits: The Linux scheduler/memory, Contiki and X.509 items are single verification conditions, not whole kernels. - how much was read: most of it read - source; notes: Dafny, C verification, and the specification-intent gap - record id: `lemmanet-2026` - MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematics (2026) - what: Introduces MATHLIBLEMMA, a four-stage LLM pipeline for mining reusable folklore lemmas missing from Mathlib. Discovery proposes statements from Mathlib seed files; an LLM judge filters mathematical plausibility; a Lean-connected formalizer repairs syntax and type errors; a prover generates proofs with Lean checking and a proof-bypass screen. The resulting benchmark contains 4,028 non-trivial type-checked statements, and the released repository contains 1,506 Lean-checked generated proofs. Across seven model/settings configurations, union Success@2 is37.39% while the best individual GPT-Reasoning model reaches19.39%; a small pilot curated six outputs for Mathlib, with three merged. Formalizes each target as a pair(S,P) in seed context C⊂L with semantic novelty, mathematical plausibility and syntactic validity requirements. Discovery is prompted to generate at least 100 candidate lemmas per seed with sorry; Judge ignores Lean syntax and returns correct/wrong; Formalizer uses Kimina Lean server diagnostics for up to10 syntax-repair trials; Prover allows an initial attempt plus two repair rounds and rejects sorry/admit/sorryAx and model-added axiom/constant/opaque declarations. Benchmark seeds 109 Mathlib source files, manually stratified into Foundational, Applied and Abstract domains; aesop-immediate instances are excluded. Main evaluation is closed-book, sequential Success@2, with full context C supplied but no retrieval. Appendix D audits 138 model-unsolved statements (107 human-proved,78%); held-out audit proves79/94 (84.0%). Appendix E documents three merged lemmas: Gronwall-bound monotonicity, restriction of a constant kernel, and almost-everywhere congruence of central moments. - limits: LLM Judge filtering is imperfect: in a 53-rejected-candidate audit,54% were false and46% valid false negatives. Judge acceptance is not a proof of truth; kernel checking establishes well-formedness, and proof checking establishes the supplied theorem, not preservation of the intended informal idea. The benchmark is generated by GPT-5.1 discovery/judging/repair and is explicitly a versioned pipeline distribution, not a model-independent census. Discovery is seed-file local and one-shot, with no repository-wide retrieval, dependency traversal or proof-attempt-driven gap discovery. The 78% residual audit samples model-unsolved instances and lets human experts use full Mathlib, AI assistance and external search. It estimates provability of that stratified sample, not the full 4,028 benchmark. The 84.0% held-out audit uses a different five-seed design and is not pooled. Closed-book Success@2 measures internalized library knowledge plus reasoning under a serial two-repair budget. It is not directly comparable to parallel Pass@k; the 8B Pass@16 appendix is explicitly a separate stress test. The proof-bypass screen excludes common incomplete or assumption-injection artifacts but does not by itself establish that a lemma is novel, well named, minimally stated or suitable for Mathlib. Authors state novelty filters are mainly syntactic and semantically equivalent restatements may remain. All Lean checks use Lean v4.25.0-rc2 and a fixed Mathlib revision through a stateless server; results may change as APIs and typeclass elaboration evolve. The reader did not rerun this environment. Union performance is an oracle union over model outputs, not one executable ensemble with a stated orchestration cost. Model comparisons use different architectures and inference settings; domain interpretations are author claims. Three of six curated submissions merged, one was rejected as redundant and two were not pursued. This is a small selection signal, not a 1,506-proof upstream acceptance rate. Costs exclude curation, local open-model serving and maintainer review. Human audits can prove statements with open-book tools, while models receive no retrieval; the human/model gap therefore mixes proof ability with access to library search and assistance. - how much was read: most of it read - - record id: `mathliblemma-folklore-lemma-generation-2026` - miniF2F-Dafny auto-active proving (2026) - title: MINI F2F-DAFNY: LLM-Guided Mathematical Theorem Proving via Auto-Active Verification - what: Translates the miniF2F mathematical benchmark into Dafny and measures how SMT automation plus LLM-generated proof hints perform in an auto-active verifier. The benchmark has 244 test and 244 validation lemmas with compact definitions.dfy and library.dfy support. Dafny translates Hoare-style specifications through Boogie weakest preconditions to Z3; ghost assertions, calc chains and lemmas guide opaque solver search. The authors validate that generated solutions preserve requires/ensures, reject assume or axiom cheating, and audit the trusted support layer: 128 of 160 originally axiomatized lemmas are proved, 14 selected lemmas are independently solved, and 142 of 201 empty-proof successes remain without definitions.dfy. LLM attempts receive up to four trajectories and three error-correction rounds. - number: 488 total problems - number: 244 test / 244 validation - number: 38.9% empty-proof test baseline - number: 62.7% Claude Opus pass@4 - limits: The translation is human-led and the mathematical support is an intentionally small axiomatized library rather than a foundational Mathlib formalization; opaque transcendental and number-theoretic properties remain trusted. Results use Dafny 4.11/Z3, 30-second attempts, eight off-the-shelf models, and no fine-tuning. Verification is brittle to assertion order/calc organization; models often confuse Dafny syntax with Lean and fail when required facts are absent. Pass rates measure generated Dafny code accepted by the validator, not independent proof objects. - how much was read: most of it read - - record id: `minif2f-dafny-auto-active-proving-2026` - Mysticeti proofs (2026) - title: Mechanized Safety and Liveness Proofs for the Mysticeti Consensus Protocol under the LiDO-DAG Framework - what: Proves a corrected Mysticeti protocol and audits Sui's deployed implementation. A counterexample lets honest participants jump over rounds without limit, so no block ever commits. The authors restrict jumping, add a catchup time GCT, and prove strong progress after both GST (the time after which message delays are bounded) and GCT. - number: 6271 new Coq/Rocq proof lines; two persons in two months (§5.5, PDF p.13) - number: 6271 lines, developed by two persons in two months - limits: Not a machine-checked proof that the whole Rust codebase matches the model. The Sui finding was true when the paper was written. It is not a claim about the current deployment. Weak liveness guarantees only that honest leader vertices and certificates get created, not that blocks commit. - how much was read: most of it read - source; notes: Mechanized Safety and Liveness Proofs for the Mysticeti Consensus Prot; notes: LiDO, LiDO-DAG, and a 2026 liveness bug - record id: `mysticeti-proofs-2026` - NetNomos (2026) - title: Making Logic a First-Class Citizen in Generative ML for Networking - what: NetNomos learns logical rules that fit network traces, using a minimal-hitting-set encoding. It filters them with GPT-4.1 and retrieval. It then uses them to constrain a generative model. - number: over 75% reference coverage within 12 hours vs DuoAI under 60% at 24 hours - number: 1.6-6.5x scalability gain - number: GPT-4.1 with retrieval precision at least 98.1%, false positives at most 0.73%, vs 82.5%/9.2% without - number: generated data violates 0.3% of learned rules on MAWI - limits: The rules are only consistent with a finite dataset. It has no protocol liveness result and no theorem about an implementation. Human hours are not stated. - how much was read: most of it read - source; notes: Making Logic a First-Class Citizen in Generative ML for Networking (Ne; notes: Making Logic a First-Class Citizen in Generative ML for Networking (Ne - record id: `netnomos-2026` - NL2VC-60 (2026) - title: From Natural Language to Verified Code: Toward AI Assisted Problem-to-Code Generation with Dafny-Based Formal Verification - what: Introduces NL2VC-60, 60 complex algorithmic problems paired with Dafny code/specifications, and evaluates seven open-weight LLMs under contextless, signature-guided and verifier-feedback prompts. uDebug tests combat vacuous proofs by requiring executable functional correctness. Tiered prompting supplies method signatures as structural anchors, then iterative Dafny compiler feedback repairs annotations/code. The study categorizes errors (compilation, specification, proof, functional) and compares verify@k. Contextless prompts almost universally fail; signatures reverse performance, and self-healing further improves models. - number: Gemma 4-31B reached 90.91% verification success - number: GPT-OSS 120B rose from 0% to 81.82% with signature feedback - number: Contextless peak verify@5 27.27% at T=0.8 - number: NL2VC-60 contains 60 problems; seven open-weight models; 11 randomly selected sets - limits: Dafny specifications and uDebug tests are trusted; tests cannot establish full semantic correctness. Dataset has 60 problems and 11 selected sets; generated solutions may overfit signatures/prompts. Open-weight model results and Dafny syntax may not transfer to Verus/Rust. - how much was read: most of it read - - record id: `nl2vc-60-2026` - NTP4VC (2026) - title: Neural Theorem Proving for Verification Conditions: A Real-World Benchmark - what: 600 verification conditions (the proof goals a verifier produces) from C/Why3 programs are translated to Lean, Isabelle and Rocq with human-written rules. Annotations are removed to make them harder, and models must prove them. - number: more than 7.5k VCs extracted, 600 selected; 300 C obligations from 17413 lines - number: more than 2400 human-written mapping rules - number: DeepSeek-Prover-V2-671B Lean pass@1/8 1.67/3.00; Minilang Isabelle 2.08/11.46; Sledgehammer 18.00; CoqHammer 5.67 - number: Why3 ATP baseline 22%; 10 min timeout - limits: Each goal is a single condition, not a whole system. It has no distributed progress. The translation is trusted. Some cases may be unprovable. The prose says 11.56% but the table says 11.46. - how much was read: most of it read - source; notes: Neural Theorem Proving for Verification Conditions: A Real-World Bench - record id: `ntp4vc-2026` - Numina-Lean-Agent (2026) - title: Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics - what: Combines a general coding agent (Claude Code with Opus4.5) and Numina-Lean-MCP tools to perform formal mathematics with autonomous tool selection. Reports12/12 Putnam2025 formal problems solved,matching AxiomProver,and a human-assisted harmonic-analysis formalization exceeding8,000 Lean lines in under two weeks. The contribution is agent/tool integration and case evidence,not a new model-training algorithm or mechanized correctness theorem for the agent. Four tool groups:Lean-LSP diagnostics/goals/search/code;LeanDex semantic theorem retrieval;informal generator/verifier;discussion partner for alternative reasoning. LeanDex rebuilds Mathlib/FLT/PhysLean under Lean4.26.0,regenerates descriptions using DeepSeekChat3.1,optimizes queries and reranks results with LLMs,with claimed monthly updates. Gemini3ProPreview generator/verifier refines up to20 times and requires three unanimous informal checks;these guide search rather than certify proofs. Subagents isolate lemmas in temporary files and merge successful verified lemmas,executed sequentially with main-agent blocking. For sustained research formalization,a dependency DAG blueprint records definitions/intermediate lemmas/final theorem and is revised from Lean feedback,human hints and external discussion. - number: 12 of 12 Putnam 2025 problems solved, budget about 50 per problem, about $300forB6andabout$1000 for A5
  • number: A5 took 2040 minutes and 3263 lines of Lean; B1 took 55 minutes and 328 lines
  • number: Brascamp-Lieb: more than 8,000 lines of Lean in under two weeks, about 70 new definitions, lemmas and theorems added by the agent
  • number: On B4, iterative refinement of informal proofs finished in 5 rounds; independent sampling failed in 10 rounds
  • limits: Formal-checking trust is relative to Lean,the exact formal statement,imported libraries and harness. Putnam uses Seed-Prover1.5 formal statements and disabled online API search,but existing library retrieval is enabled. No artifact or proof certificate was independently checked by this reader;supplied AppendixA statement ends by sorry and AppendixB lemma has no proof body. Twelve Putnam problems plus selected six-problem ablations are small evaluations. No multiple-run variance,random seeds,confidence intervals or standardized hardware/API latency accounting is provided. Inter-system timing/line comparisons are reported values,not demonstrated matched-compute causal comparisons. Default budget approximately$50/problemisraisedto$1000 A5 and300 B6;authors call these relative effort rather than exact accounting. Main headline is success under unequal per-problem resources,not fixed-cost12/12. Sequential execution avoids parallel search but does not imply uniform cost or faster overall solving. The research case is collaborative:one mathematician,one Lean expert and agent use human intent review,inline hints,priorities,type conversions and blueprint edits. >8,000 lines and≈70 new objects are output size,not a human-effort reduction estimate. No human-only control,total token count or complete task-time accounting is reported. LLM informal verification and discussion are fallible search guidance. Three agreeing checks by the same model family are not independent formal certificates. Lean diagnoses formal acceptance;human review is still needed when the agent changes mathematical assumptions or theorem statements. Tool ablations couple informal proving and discussion;the reported triples do not isolate each tool’s causal contribution. AppendixD separately claims A5/B6 fail without discussion,but supplies no corresponding numeric table,budget or repeated-run data. Invocation frequency alone cannot establish necessity. Base-model ablation has Opus6/6,DeepSeekV3.2 3/6,KimiK2 zero shown successes on six problems. Openness/interchangeability is an architectural claim;published tools still rely on proprietary Claude/Gemini components in best configuration. No Rust/Verus/TLA+ or distributed-system verification evaluation. - how much was read: most of it read - source; notes: ## Numina-Lean-Agent (2026) - record id: `numina-lean-agent-2026` - Open-Source LLM-Driven Formal Verification: A Multi-Agent Pipeline for RTL Repair (2026) - what: An LLM pipeline fixes a bug in a small hardware design (RTL). A solver checks each fix and returns a counterexample trace when it fails; the trace goes back to the LLM until the fix is proved for all reachable states by k-induction (a proof that a property holds at every step, built from a base case and an inductive step). - number: 6 hardware benchmarks, 5 runs each, GPT-4o, temperature 0, at most 10 iterations - number: alu repaired in 5/5 runs, 2 iterations, 16.5 +/- 3.0 s - number: counter, arbiter, axi_lite_slave, uart_tx: 0/5 each; fifo: 0/3 valid runs (2 excluded for API quota) - number: failing runs always hit the 10-iteration cap - limits: Six small modules, one hand-injected bug each, one model; the author calls it a feasibility study. Properties are bug-checking assertions written by the LLM from a natural-language spec, with no check that they capture the spec. - how much was read: most of it read - source; notes: ## RTL repair with open-source formal tools (2026) - record id: `open-source-llm-driven-2026` - OProver (2026) - title: OProver: A Unified Framework for Agentic Formal Theorem Proving - what: A Lean 4 prover that retries failed proofs using similar verified proofs and the compiler's error text, and is trained on those retry traces. It also releases OProofs, a corpus of Lean statements, verified proofs and repair traces. - number: OProver-32B Pass@32: MiniF2F 93.3, ProverBench 58.2, PutnamBench 11.3, MathOlympiad 22.8, ProofNet 33.2 - number: removing compiler feedback and retrieval drops PutnamBench for the 32B model from 11.3 to 5.9 - number: corpus: 1.77M statements, 6.86M verified proofs, 164K trajectories with repair (the body text elsewhere says about 1.5M and 5.0M) - number: MiniF2F Pass@32 rises across training rounds: 8B 79.5 to 91.8, 32B 84.7 to 93.3 - limits: Math theorems only, with statements given; the paper has no separate limitations section. Baselines are run under their own protocols and some numbers are copied from other papers. - how much was read: most of it read - source; notes: OProver (2026) - record id: `oprover-2026` - OSVBench (2026) - title: OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification - what: A benchmark in which an LLM writes a Python/Z3 specification of a syscall from a description. The spec is scored by whether it detects the same injected errors in code variants as the reference spec does. Declaration specs stay fixed. - number: 245 tasks from 49 system calls, 5 variants each (1 correct, 4 with injected bugs) - number: 12 LLMs; best pass@1 55.10% (Doubao-1.5-pro); five zero-shot models score zero - number: five-example prompts use 20,000-30,000 tokens - number: 245 tasks from 49 system calls (each: one correct and four vulnerable variants) - limits: It covers functional and safety properties only, with no liveness. The LLM is given a description, assumptions and an abstract kernel model. - how much was read: most of it read - source; notes: OSVBench: executable specifications with a fixed external oracle; notes: OSVBench: executable specifications with a fixed external oracle - record id: `osvbench-2026` - P Verifier tutorial (2026) - title: P Verifier two-phase commit tutorial - what: The P documentation tutorial walks through a simplified two-phase-commit model, specification and test drivers, then points to the proof backend. The page is a practical teaching path from a protocol description to checked properties. - limits: Simplified protocol and tutorial scope; not a production proof. - how much was read: most of it read - source - record id: `p-verifier-tutorial-2026` - Paxos-SB in DistAlgo (2026) - title: Specifying Paxos for System Builders: Pseudocode Made Executable - what: It transcribes detailed Paxos-SB pseudocode into runnable DistAlgo. At runtime it checks agreement, validity and timeouts. It finds two liveness bugs in the published pseudocode: a non-leader, and both survivors not counting their own vote. - number: Table 1: 598 lines total; about 370 excluding comments/setup; core about 200 - number: original C about 7000 lines; pseudocode about 330 lines in 13 figures - number: 3 servers, 2 clients, 100,000 requests each in under 170 s; latency under 1.7 ms; about 10x C runtime - limits: These are runtime checks, not proofs. Humans supplied initial values and helpers, and no LLM workflow is reported. The recovery procedure is not implemented separately. - how much was read: most of it read - source; notes: Specifying Paxos for System Builders: Pseudocode Made Executable - record id: `paxos-sb-in-distalgo-2026` - PaxosLease Revisited: A Checked Model of Diskless Distributed Leases (2026) - what: It has bounded checking in TLA+/TLC, six TLAPS arithmetic lemmas, and Python witnesses for the PaxosLease diskless lease protocol, with three acceptor changes. LLMs produced much of the artifacts, under line-by-line human review. - number: Table 1 traces of 27, 36, 18, 36, 43 states - number: Table 2: 491,037 base states, 37,425,056 crash/restart, 76,749,192 at quarantine; completed runs 2,179,760,458 and 2,411,951,538 states - number: two larger searches stopped without violation at 3.86 and 4.46 billion states - number: more than 40 hours on a 4-core Xeon, 64 GB - limits: It covers safety only, with at most two proposers, three acceptors and three ballots. It does not show that the C++ code refines the model. The shipped binaries were not run. Searches that were stopped are not exhaustive. Fingerprint collisions qualify the largest runs. - how much was read: most of it read - source; notes: PaxosLease Revisited: A Checked Model of Diskless Distributed Leases - record id: `paxoslease-revisited-a-checked-2026` - ProofWala (2026) - title: ProofWala: A Framework for Multilingual Proof Data Synthesis and Theorem-Proving - what: Provides itp-interface and ProofWala, a shared infrastructure for extracting and executing Lean/Rocq tactics, training proof-step models, and searching cloned proof environments in parallel. Controlled 220M-parameter model experiments find significant multilingual gains on Mathlib and CategoryTheory adaptation, with smaller nonsignificant pass@5 gains on CompCert, MathComp and GeoCoq. Sections 2–3 formulate proving as transitions between finite sets of obligations, implement Lean instrumentation inside the elaborator, export declaration/dependency graphs to SQLite, and retain reusable environment pools keyed by current proof state. Beam-search pseudocode generates, concurrently executes, and prunes tactic candidates; annotated trees contain only assistant-accepted transitions. Sections 4–5 document theorem-level corpus splits, identical prompt serialization without assistant/repository labels, fixed-step CodeT5 training, pass@1–5 and paired bootstrap tests, analogous Lean/Rocq category-theory examples, search-tree size/degree and tactic-length/runtime tables, MiniF2F transfer and CPU scaling. Section 3.1 explains why preliminary Isabelle/PISA integration is too expensive to scale. - number: 442,607 training state/tactic pairs; 76,997 training theorems; 270M tokens - number: 220M parameters; 34,000 max updates; batch 128 - number: Lean pass@5 28.25→30.58%, p=.018 - number: CategoryTheory pass@5 45.54→53.47%, p=.008 - limits: Empirical scope is tactic-based Lean and Rocq with CodeT5-Base (220M), not a frontier-LLM or specification-generation benchmark. Lean support is reported as 4.15.0–4.24.0 with a compatibility layer; detailed Rocq backend support is 8.10–8.18. CompCert/Mathlib reuse prior splits; other corpora use theorem splits, so structurally analogous theorems across languages can remain in training (explicitly analyzed in Table 4). No stronger contamination audit is demonstrated. Search uses 600-second timeouts, 1200 seconds for GeoCoq. Isabelle support is preliminary and requires tens of gigabytes per PISA instance. Gains outside Lean and adapted CategoryTheory are not statistically significant; multilingual MathComp and GeoCoq pass@1 are lower than monolingual. Proof acceptance inherits assistant correctness; this paper does not prove a distributed program or generate Verus/TLA+ specifications. - how much was read: most of it read - source - record id: `proofwala-2026` - Re:Form (2026) - title: Re:Form: Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny - what: It trains models with supervised data and reinforcement learning to write Dafny annotations. It adds a reward that checks two things: the generated contract accepts every input the reference accepts, and it promises at least the reference's outputs. This stops reward gaming with weak specs. - number: 20,000-program collection, 3,000 supervised and 4,500 RL examples, 512 in-domain and 300 DafnyComp tests - number: 14B RL verification 14.0% vs SFT 8.3% - number: out-of-domain spec-superiority 0.077 vs 0.047 - number: 14B training: 64 GPUs for about 36 hours - limits: The references are themselves autoformalized, so the implication does not certify fidelity to human intent. Mostly sequential functions. - how much was read: most of it read - source; notes: Re:Form: Reducing Human Annotations in Scalable Formal Software Verifi - record id: `re-form-2026` - RLVR-DafnyBench (2026) - title: RLVR-DafnyBench repository - what: RLVR-DafnyBench is a Docker/uv pipeline for reinforcement learning from verifier rewards on DafnyBench, with scripts for data download, training, evaluation and tests. - limits: README describes a pipeline; no training or evaluation was rerun. - how much was read: most of it read - source - record id: `rlvr-dafnybench-2026` - Scope (2026) - title: Detecting Inconsistencies in Arm CCA's Formally Verified Specification - what: Translates the Arm Realm Management Monitor spec into a Verus model checked by an SMT solver. It cross-checks different descriptions of the same behavior (conditions, effects, state diagrams, dependencies). It needs no reference implementation. - number: 35 previously unknown bugs, all confirmed by Arm (of 38 candidates) - number: 61.90% precision on one evaluated version/subset; do not mix with the 35 count - number: about 11 months of effort, chiefly tool implementation - limits: Consistency does not mean the spec is adequate: an omission shared by every view is missed. It also misses properties that are stated nowhere else. Its bit and byte semantics are incomplete, and uninterpreted functions cause false alarms. It has no benchmark for liveness over time. - how much was read: most of it read - source; notes: Specification errors and auditing; notes: Scope; notes: Appendix B: Concrete systems and theorem scope - record id: `scope-2026` - SL-VC and SPLIT (2026) - title: SL-VC: A Benchmark and Automated Framework for Separation Logic Verification Condition Proving - what: Introduces SL-VC, 289 separation-logic verification conditions from textbook data structures and real C system code, and SPLIT, an LLM-friendly Rocq tactic library plus two-stage spatial/pure proof workflow. SPLIT decomposes mixed entailments into spatial elimination and independent pure goals, using explicit tactics such as split_pure_spatial, cancel, sep_apply and pure_solve with verifier-driven repair. The benchmark includes linked lists, merge sort, trees, LiteOS circular lists and mini-gmp; the full 288-VC evaluation and 207-VC core subset separate benchmark construction from prover comparisons. CoqHammer is evaluated as an alternative pure-stage solver, and failures concentrate in arithmetic-heavy mini-gmp and recurrence-like goals. - number: 289 VCs total; 288 evaluation and 207 core subset - number: SPLIT 66.0% at k=10 and 14.2% at k=0 on full set - number: human-friendly baseline 43.4% at k=10 and 4.5% at k=0 - number: core subset SPLIT 73.4% at k=10 versus PALM 14.0% and Rango 7.7% - limits: Programs and annotations come from QCP and are selected to make self-contained VCs; difficult algorithmic/refinement and complex data invariants are outside the benchmark. SPLIT depends on goals admitting spatial/pure decomposition and on the Rocq/QCP tactic/lemma library. Pass rates measure verified scripts under finite repair budgets, not complete verification of whole C systems. - how much was read: most of it read - - record id: `sl-vc-and-split-2026` - Software is infrastructure: failures, successes, costs, and the case for formal verification (2026) - what: A chapter for teaching and policy. It collects secondary industrial reports on the costs of software failures and the benefits of formal verification (Philips ASD, 4SECURail, Dassault, AWS, CCF). - number: Philips ASD: four-fold better quality, three-fold productivity - number: 4SECURail projected benefit/cost 5.05 - number: Frama-C handled 85% of assertions on about 50 KLOC blocks at Dassault; a person-month saved per release - number: six subtle bugs found binding CCF C++ traces to TLA+ - limits: The reports are heterogeneous and secondary. There is no common benchmark or controlled estimate. It proves no new theorem. - how much was read: most of it read - source; notes: Software is infrastructure: failures, successes, costs, and the case f - record id: `software-is-infrastructure-failures-2026` - SoK consensus (2026) - title: Systematization of Knowledge: Formal Verification of Consensus Protocols - what: A survey that sorts consensus verification into model checking, deductive proofs, code proofs and hybrids. - number: over 20 verified consensus protocols - number: only about 30% of surveyed works address liveness - number: Matrix covers 20+ protocols in abstract and over 30 in conclusion - number: Only ≈30% of surveyed works address liveness (§5) - limits: It gives no search date range and no count of how many papers were considered, so the reported fraction is not an unbiased census. Table 1 entries need checking against the cited papers. - how much was read: most of it read - source; notes: Systematization of Knowledge: Formal Verification of Consensus Protoco - record id: `sok-consensus-2026` - SpecCoder (2026) - title: Teaching Code LLMs to Reason with Intermediate Formal Specifications - what: Fine-tunes code LLMs to insert runnable assertions at internal program points (not just final postconditions), keeping only assertions that hold on correct runs and fail on buggy mutants. Also releases HumanExec, 150 Codeforces tasks with real buggy submissions. - number: prompted Qwen2.5-Coder-32B: 0.8989 syntax validity, 0.6313 correctness, 0.2604 completeness (Table I) - number: after SpecCoder, Qwen2.5-Coder-14B inline completeness 0.1989 to 0.7713 and validity 1.0000 (Table II) - number: average inline completeness over three model sizes 0.2112 to 0.7459 - number: HumanExec: 150 tasks, 48-279 tests each (mean 79.53) - limits: Sequential, single-run Python programs from competitive programming; no distributed or concurrent systems and no proofs, only assertions checked on test runs. Completeness is measured against available faulty runs, not all possible bugs; training data came from a teacher LLM plus test-suite filtering. - how much was read: most of it read - source; notes: ## SpecCoder and HumanExec (2026) - record id: `speccoder-2026` - Specula (2026) - title: Specula: Scaling formal specifications for autonomous model checking of system code - what: Specula writes TLA+ models and properties from system source code. It checks observed traces against the model and repairs disagreements. It then model checks for violations and tries to reproduce each one in the real code. - number: 48 open-source systems; 249 reproduced bugs; 89 reported to maintainers; 68 confirmed and 24 fixed in that subset - number: only 0.90% of generated properties are liveness; 99.10% safety - number: 200 bugs via model checking, 49 during code reading/modeling - number: median 3.69 hours and 57 per system across 48 systems (§5.3)
  • limits: Model checking plus reproduction gives concrete failures, not proofs for all executions. Its coverage of liveness is weak. The results pool several versions, and only 14 systems used the latest version.
  • how much was read: most of it read
  • source; notes: Specula: close the model–code loop before calling a violation a bug
  • record id: specula-2026
• Specy (2026)
  • title: Specy: Automatically Learning Specifications for Distributed Systems from Event Traces
  • what: It learns event-based formulas from traces of distributed systems (P models or logs). It builds monitors from them and checks fresh traces to reject guesses that are too strong. For six deductive-verification experiments, the learned formulas were translated by hand.
  • number: all 43 goal specs recovered, 28 (65%) without help, 15 needed guidance
  • number: 14 benchmarks incl. 3 industrial, 10,000 traces each
  • number: about 9% of extra learned formulas fail on fresh traces, no designated goal formula does
  • number: pruning speeds up by geometric mean 14.8
  • limits: The only guarantee is consistency with the supplied traces. Hard properties need predicates, event choices or ghost state. The deductive results cover selected safety properties of P models. There is no refinement to production code.
  • how much was read: most of it read
  • source; notes: Specy: Learning Specifications for Distributed Systems from Event Trac; notes: Specy: Learning Specifications for Distributed Systems from Event Trac
  • record id: specy-2026
• Sprout Network-Parametric Implementability (2026)
  • title: Implementability of Global Distributed Protocols modulo Network Architectures
  • what: Characterizes implementability of global message-passing protocols across peer-to-peer FIFO, senderbox, mailbox, monobox and bag architectures using network-parametric coherence conditions and operational buffer axioms. The paper proves architecture relationships, co-NP-completeness for finite protocols, and symbolic decision procedures. The network-parametric Sprout(A) tool instantiates the conditions for five architectures and checks existing plus architecture-separating benchmarks through MuVal.
  • number: Finite-protocol implementability is co-NP-complete for all five architectures
  • number: Sprout(A) runs on a 2024 MacBook Air M3/16 GB, averaged over 10 runs; average times p2p 6.7 s, sb 5.8 s, mb 7.6 s, monob 7.1 s, bag 12.6 s
  • number: Table uses 15 s per-muCLP timeout; bag is generally slowest due to relaxed ordering
  • limits: Soundness/completeness of Sprout(A) is relative to MuVal, which may diverge because symbolic-protocol implementability is undecidable. Evaluation uses finite benchmark suites, a 15-second timeout per muCLP instance, and manual inspection for selected results; performance varies with message-buffer state-space size.
  • how much was read: most of it read
  • record id: sprout-network-parametric-implementability-2026
• Survey of Consensus Protocol’s Formal Verification (2026)
  • what: This is a 42-page survey of formal verification of consensus protocols. It has tables of papers, targets, properties, tools and experimental parameters.
  • number: Tables 3-6 collect papers
  • limits: Some explanations are too broad, for example ‘SMT is NP-complete’ and ‘probabilistic checking abandons complete search’. Use the primary studies. The survey states no search protocol.
  • how much was read: most of it read
  • source; notes: Survey of Consensus Protocol’s Formal Verification
  • record id: survey-of-consensus-protocol-2026
• SysMoBench (2026)
  • title: SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems
  • what: A benchmark of eleven concurrent and distributed systems. An AI writes a TLA+ model of each. The model is scored on syntax, execution, behavioral match to the code, and expected properties.
  • number: source sizes from 175 lines (PGo dqueue) to 5,360 (ZooKeeper leader election)
  • number: Etcd Raft with basic Sonnet agent: 7.69% conformance, 69.23% invariant correctness, 100% syntax
  • number: liveness properties violated 41.9% vs safety 8.3% (§5, Figure 4c)
  • number: each agent run five times, best output evaluated, at most three repairs
  • limits: Covers only the finite executions and configurations supplied. A model can match buggy code and still fail the desired guarantee. The evaluator uses an LLM for some mappings.
  • how much was read: most of it read
  • source; notes: SysMoBench
  • record id: sysmobench-2026
• SysSpec / SpecFS (2026)
  • title: Sharpen the Spec, Cut the Code: A Case for Generative File System with SysSpec
  • what: Humans write structured specs for 45 modules, with pre/postconditions, invariants and locking. Agents generate C code, and an LLM reviews it. The result is SpecFS, an in-memory FUSE file system.
  • number: about 4,300 C lines, 45 modules
  • number: 754 xfstests, 64 fail from unsupported features
  • number: Gemini-2.5-Pro and DeepSeek-V3.1 reach 100% module accuracy vs 81.8% best baseline
  • number: extent support 4.5 vs 1.5 hours; directory rename 13 vs 2.4 hours (manual vs SysSpec)
  • limits: The verification stage is an LLM, not a theorem prover. The evidence of correctness is tests and inspection. Crash consistency is out of scope. The timings cover specific tasks only.
  • how much was read: most of it read
  • source; notes: SysSpec / SpecFS: sharp human specifications, empirical code checking; notes: SysSpec / SpecFS: sharp human specifications, empirical code checking
  • record id: sysspec-specfs-2026
• Timers (2026)
  • title: Verifying First-Order Temporal Properties of Infinite-State Systems via Timers and Rankings
  • what: Turns a claim about time into a termination problem by adding timers that predict when formulas become true. A human supplies the model, the claim, the implicit ranking, the invariant and finite-set approximations. Z3 checks the generated conditions.
  • number: 17 tasks in Table 2, e.g. Ticket: 2 s, 20 invariant conjuncts, 256 proof tokens
  • number: Paxos: 101 s, 238 tokens; ABP: 4 s, 539 tokens; HRB-Correct: 135 s, 99 tokens; Dijkstra k-State: 64.8 s, 213 tokens
  • number: Ivy proof sizes for six tasks: 407, 521, 860, 446, 593, 748 tokens (Ticket, Paxos, ABP, HRB-Correct, HRB-Relay, Ackermann)
  • number: runtime is Z3 validation on Apple M1 Pro, Z3 4.13.3
  • limits: The guarantees cover first-order transition-system models, not deployed code. Rankings and invariants are not discovered automatically. Sizes count proof tokens, not human effort.
  • how much was read: most of it read
  • source; notes: Verifying First-Order Temporal Properties of Infinite-State Systems vi
  • record id: timers-2026
• TLA+-Bench (2026)
  • title: TLA+-Bench: An Execution-Grounded Benchmark and Dataset for Natural-Language to TLA+ Specification Generation
  • what: TLA-Bench has TLA+ specs, each with a model-checker config and English descriptions. A model reads the description and writes a spec. TLC (the TLA+ model checker) runs it against the reference config. A pass means the spec parses and no listed property is violated.
  • number: 403 model-checked gold and 897 parse-only silver specs from 13 public repositories
  • number: on one fixed set of 300 outputs the correct rate moves from 10.0% to 1.7% depending on grading choices; 18.7% to 1.7% if the model is told the config’s names
  • number: best model is correct 16% of the time by default and 26% given the interface names; open models at most 1%; best model parses 87%
  • number: 12 of 30 passes meet the non-vacuity bar, so the plain correct rate overstates capability about 2.5 times
  • limits: The descriptions are written by a model, not by practitioners. Composition and refinement are under-represented. The paper makes no claim that controls for contamination. A pass is exact only relative to the reference config. It does not prove that the spec reproduces the reference behavior. The notes’ later pass says the benchmark creators wrote the descriptions and the grading oracle.
  • how much was read: most of it read
  • source; notes: TLA+-Bench
  • record id: tla-bench-2026
• TLA+ Examples repository (2026)
  • what: This repository is a corpus of TLA+ specifications and models, with CI and metadata indicating beginner examples, PlusCal variants and TLC, Apalache or TLAPS use. Its README also labels models that intentionally fail on deadlock, safety, liveness or assumptions.
  • number: at commit 6a7b26a: 81 case directories, 424 modules, 233 model configurations, 73 modules with proof metadata
  • limits: Repository metadata and CI status can change; this pass did not run every model.
  • how much was read: most of it read
  • source
  • record id: tla-examples-repository-2026
• TLA-Prover benchmark (2026)
  • title: TLA-Prover repository benchmark documentation
  • what: This repository documents a harness for checking whether LLMs produce and repair TLA+ that pass SANY and non-vacuous TLC. Its README freezes corpus/holdout gates and reports audited model results, including false passes caused by weakened or deleted properties.
  • number: 206 specs in the corpus; 30-spec holdout
  • number: Post-audit model-only repair 166/206; W4-diamond-gold post-prompt-fix 16/30 pass@32
  • limits: Repository status is dated and reports were not rerun; results are self-reported in repository artifacts.
  • how much was read: most of it read
  • source
  • record id: tla-prover-benchmark-2026
• TLAi+Bench (2026)
  • title: TLAi+Bench repository
  • what: TLAi+Bench evaluates LLMs on natural-language-to-TLA+ puzzles and real-world scenarios with reference specifications and GenAIScript generation/verification utilities.
  • number: Nine named puzzle categories/files are listed in the README; six gold reference modules are shown.
  • limits: The current repository emphasizes puzzles and examples; repository status and task coverage can change.
  • how much was read: most of it read
  • source
  • record id: tlai-bench-2026
• TLAPS-Bench (2026)
  • title: TLAPS-Bench repository
  • what: TLAPS-Bench asks AI agents to prove invariants of TLA+ protocol/system specifications in a sandboxed runtime. Its current problem set includes ZooKeeper, etcd and HashiCorp Raft, MongoDB transactions and other protocols.
  • number: Current problem set: 9 specs and 57 invariants
  • number: Retired/next lists include Byzantine Paxos, broadcast, Paxos, 2PC, termination detection and WAL protocols.
  • number: 9 specs and 56 proof targets (FLASH 15, ZooKeeper 9, etcd Raft 8, HashiCorp Raft 6, others)
  • limits: The README says retired proofs are available by contact and current runs focus on a few hard tasks; no agent run was reproduced.
  • how much was read: most of it read
  • source
  • record id: tlaps-bench-2026
• Traceable and Verifiable Software Requirements: A Synthesis of AI-Enabled Formal Methods (2026)
  • what: A survey plus an experience report. The authors ran Frama-C (RTE, EVA, WP with several solvers) and PathCrawler on C functions to see what automation gives for requirements.
  • number: tutorial experiment 36 programs; five-program subset goals proved 6/6, 5/7, 6/8, 4/8, 13/13
  • number: triangle classification: 20 goals, Alt-Ergo/CVC5 prove 15, Z3/CVC4 prove 13
  • number: fifty base programs (150 instances) vs Table 4 files 52+60+53=165
  • number: PathCrawler succeeds on 4 of 20 files, then 15 of 20 after adaptation; quicksort: 11 cases from 110 paths, 99 infeasible
  • limits: The accounting in Table 4 conflicts with the prose. The ResearchGate version is unreadable. An oracle was unavailable. No LLM starts from a vague request.
  • how much was read: most of it read
  • source; notes: AI-Enabled Formal Methods for Verifiable Software Requirements: A Surv
  • record id: traceable-and-verifiable-software-2026
• TrojanSpec-Bench (2026)
  • title: TrojanSpec-Bench: An Adversarial Benchmark for LLM-Based Formal Specification Elicitation
  • what: Treats the LLM that turns English requirements into a formal spec as a possible attacker. Builds 1,024 cases in Dafny, Lean 4 and Verus where a spec is subtly wrong yet still accepted by the verifier together with a bad implementation, then tests six detectors for such specs.
  • number: 1,024 admitted cases of 1,798 generated (57.0%): Dafny 319, Lean 4 417, Verus 288
  • number: four attack types admitted: implementation leak 377/450, vacuity 348/450, domain restriction 187/450, predicate swap 112/448
  • number: detector asking an LLM four yes/no questions and flagging when at least 2 fail: F1 0.967, false-positive rate 0.068, recall 1.000
  • number: single LLM-judge vote baseline: F1 0.871, false-positive rate 0.297
  • limits: Same model family (Claude Sonnet 4.6) generates most trojans and judges them; the LLM detector was not tested against adaptive attackers; ‘honest’ labels come from the verifier, not from humans; attack mix reflects the generator, not real-world frequency.
  • how much was read: most of it read
  • source; notes: TrojanSpec-Bench (2026)
  • record id: trojanspec-bench-2026
• Veil issue 26 (2026)
  • title: With veil.smt.trust false, a transition’s obligation errors but #check_invariants still prints success
  • what: The issue reports that Veil can print a checkmark even when an SMT obligation errors under the no-trust setting.
  • number: Issue reproducer shows an obligation error paired with successful-looking text output.
  • limits: Open issue; pre-release/main revisions are named in the report.
  • how much was read: most of it read
  • source
  • record id: veil-issue-26-2026
• VeriContest (2026)
  • title: VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation
  • what: A Verus benchmark that scores spec, code and proof separately and end to end. Spec preconditions need proofs of equivalence in both directions. Postconditions are run on valid and invalid pairs. Proof-only mode rejects edits to the AST or contract and uses –no-cheating.
  • number: 946 tasks (690 LeetCode, 256 Codeforces), 91 manual seeds
  • number: median 276 positive and 2,670 negative examples per item
  • number: GPT-5.5: 48.31% spec, 92.18% NL-to-code, 13.95% proof, 5.29% end-to-end
  • number: 946 problems, median 276 positive and 2,670 negative tests each
  • limits: Compatibility with one reference behavior is not general soundness. Negative examples the model generates itself may share its misunderstanding. Manual review is not shown to be blind. Possible contamination and selection bias. Algorithm tasks only, with no infinite distributed executions.
  • how much was read: most of it read
  • source; notes: VeriContest
  • record id: vericontest-2026
• Verification of the Garsia–Wachs Algorithm (2026)
  • what: Dafny formalization of naive and optimized Garsia–Wachs variants returns an alphabetic binary tree of minimum weighted leaf-depth cost. Extracts a direct recursive one-phase construction from the Kingston–Knuth correctness proof: recursively solve a shorter reordered weight sequence, then split the proof-selected leaf to restore an optimum tree for the original order. Both present implementations remain quadratic; logarithmic data-structure refinements are proposals. Records unlabeled-tree representation and separate weight semantics, exact pair/insertion conditions, constructive W/X/Y/Z proof chain, zero-sum/nonnegative cyclic-shift proof, recursive and mutual-recursive termination arguments, overlapping2-descending interval invariant, and concrete performance bottlenecks. Distinguishes checked functional correctness from unimplemented asymptotic refinement and unmeasured solver/runtime behavior.
  • limits: §§1/3–4, pp.1/6–7/13: mathematical introduction allows nonnegative weights; actual algorithm contracts use nonempty seq, so verified inputs are nonnegative integers, not arbitrary real frequencies. Cost helper accepts integer sequences, but optimization theorem is stated for naturals. Result is an unlabeled full binary-tree shape with exactly input length leaves; associating weights by inorder position gives original alphabetic order. No empty input or negative-weight algorithm contract. §3, pp.6–7: Tree=Leaf|Node(left,right), no stored leaf weights. Levels and CostSeq zip a separately supplied weight sequence with leaf depths, then sum. DeleteLeaf requires internal root and a valid leaf index; removes leaf and collapses parent into sibling. SplitLeaf replaces selected leaf by two children. This avoids relabeling many leaves during exchange proofs. Split cost identity combines adjacent weights before splitting and adds their sum. §1/§4.2, pp.2/8–9: select least k in1..n with k=n or q[k−1]<=q[k+1], equivalent to first locally minimal adjacent-sum pair. Earlier positions satisfy q[i−1]>q[i+1]. Insert combined q[k−1]+q[k] at greatest j<k with j=0 or q[j−1]>=sum; all j<=i<k−1 have q[i]<sum. Strict/nonstrict inequalities and leftmost choice belong in proof preconditions, including zero and tied weights. §4.1, pp.7–8: W says if q[k−1]>q[k+1], every optimum tree has l[k]<=l[k+1]; equality of weights gives some optimum with this relation. Exchange lemma applies to any suitable tree with l[k]>l[k+1], deletes leaf k and splits its successor, reducing cost at least q[k−1]−q[k+1] and making the pair siblings. Nonnegative subtree weights support this comparison. §4.2, pp.8–9: X first constructs an optimum tree with nondecreasing levels through k and l[k−1]=l[k]. Claim2 forces l[j]>=l[k]−1 by contradiction using delete/split and negative cost change; proof separates adjacent/nonadjacent exchanged leaves. Final witness satisfies l[j]=l[k]−1, or l[j]=l[k] with j a left child. Claim1 and Statement return witness trees, not only existential statements; Claim2 establishes a property of its supplied tree. §4.3, pp.9–10: Y transforms the specially selected X witness, not an arbitrary optimum tree. Deleting the appropriate sibling leaf produces shorter-sequence tree of exact cost c−q[k−1]−q[k], hence minimum shorter cost cprime<=that value. For case(b) delete j or j+1; for case(a) delete first index s at level l[k] (or sibling s+1). Level equalities establish unchanged outside costs and the combined coefficient. §4.4, pp.10–12: Z starts from any optimum shorter tree and chooses greatest s1<k at same level as its leaf j. SplitLeaf(tprime,s1) restores original input length and has cost<=cprime+combined weight; combine Y to get equality and optimality. Intermediate rotation of weights among equal-depth leaves preserves cost. CyclicShift2 proves the original order is no costlier than shifted order by telescoping coefficients and summing strictly positive weight differences times nonnegative level differences. No parity case split needed. §4.5.1, pp.12–13: original bottom-up nonalphabetic phase needs induction that its output is level-equivalent to an optimum alphabetic tree; author points out this reasoning was implicit in Kingston/Knuth. New variant recursively constructs the shorter optimum directly, then applies Z. It avoids intermediate nonalphabetic tree and level-equivalence postconditions; does not claim the original algorithm was mathematically incorrect. §4.5.2, pp.13–14: naive GWAlgorithm decreases sequence length, basecase one Leaf. LeastNumber and GreatestNumberLe are executable linear searches; k and j exist by final-index and zero fallbacks. Recurse on length−1 sequence, find greatest s1 between j and k−1 with equal level, split. Postcondition certifies minimum cost among all shapes with correct leaf count, not uniqueness of optimum shape. §4.5.3, pp.15–17: optimized GWAlgorithm takes1<=r<=length with2-descending prefix; initial r=1. Main method decreases lexicographic(length,length−r), Combine decreases(length,length−r,|ks|). Main passes overlapping interval lengths ks=[r] at a candidate pair. TwoDescendingIntervals recursively joins prefixes of length>=2 with exactly one shared position; each last pair is potentially locally minimal. Merging interval lengths removes repeated scans; sequence-shortening recursion preserves the semantic optimum postcondition. §4.5.3, pp.15–16: FindJ first binary-searches neighboring-weight maxima to exploit2-descending rather than globally descending weights. Combine handles no-longer-minimal pair by advancing main scan or merging first two intervals; actual reduction updates ks differently for j<=1/j>1 and k=1. It selects s1 with binary search only after proving monotonic equal-level prefix properties of shorter optimum. Published code omits ghost assertions/lemma invocations, which author says are numerous. §4.5.4, p.17: assuming array-backed immutable sequences and constant indexed access, FindJ logarithmic but concatenation/slicing copies linearly, computing leaf levels needs traversal/precomputation, and SplitLeaf traverses a simple tree. Present full algorithm O(n²). Rank-augmented AVL sequence structures, paired-neighbor weights, efficient leaf-position access or deferred output-tree construction could support O(nlogn); these are proposed refinements, not verified implementations in this paper. Complexity argument is prose, not a supplied formal cost theorem. §5, pp.17–18: about3600DafnyLOC for both versions, author report. Many calc hints and trial-and-error solver-resource tricks needed to obtain consistent verification; no verification time, timeout distribution, machine, solver/version configuration, runtime comparison, large workload or annotation ratio reported. One explicit compiled-example claim and explanatory traces support illustration, not a performance benchmark. §§2/5, pp.5/17: Dafny sends obligations through Boogie to Z3. Automation discharges index/slice safety, with assertions/lemma calls as hints. Paper does not provide kernel proof terms, verified compiler theorem, SMT trust audit or independently executed artifact. LLM declaration explicitly rules out GenAI production of article/artifact; this is a human-written verification case, not an agent benchmark.
  • how much was read: most of it read
  • source
  • record id: verification-of-the-garsia-2026
• Verified IronKV issue 8 (2026)
  • title: recieve_packet spec is underconstrained
  • what: The issue says a receive transition postcondition leaves store and delegation fields unconstrained, so an implementation rewriting them could still verify. TLC found the gap when exported protocol successors could not be enumerated.
  • number: The issue reports a two-state TLC trace exposing the store-unchanged predicate failure.
  • limits: Open issue; report concerns a specific repository revision.
  • how much was read: most of it read
  • source
  • record id: verified-ironkv-issue-8-2026
• Verifying Graceful Degradation in a Distributed Malware-Detection System with SPIN (2026)
  • what: The authors model, in Promela, the endpoint side of a Bitdefender-style malware-detection system that falls back from a primary server to a legacy server to a weaker local scan, and check six properties written in temporal logic (five safety, one liveness) plus deadlock freedom with the SPIN model checker.
  • number: All six properties hold on the core 1 endpoint x 1 scan model: 6,168 states stored, depth 101, under 1 second (Table 2)
  • number: Streaming model (1 endpoint, 2 scans): 314,527 states; concurrent model (2 endpoints, 1 scan): 22,177,616 states (Table 4)
  • number: 2 endpoints each with 2 scans ran out of memory, so no result
  • number: 4 reachability trap claims all violated, showing the checked behaviors occur (Table 3)
  • limits: Assumes the transport is reliable (no loss or duplication), deadlines are untimed nondeterministic choices, and two detection contracts: the full servers always give the right verdict (A1) and the reduced scan never flags a benign file (A2). Finite instances only; unbounded endpoints and scans need parameterized techniques left to future work. The model is hand-abstracted from the production system and not tied to its code.
  • how much was read: most of it read
  • source; notes: ## Graceful degradation in a malware-detection system with SPIN (2026)
  • record id: verifying-graceful-degradation-in-2026
• VERINA (2026)
  • title: VERINA: Benchmarking Verifiable Code Generation
  • what: A Lean benchmark that scores generation of code, spec and proof separately and in combination. It checks specs by trying to prove implications against a reference spec, then falls back to tests. A test pass without a proof returns ‘unknown’.
  • number: 189 manually curated Lean problems
  • number: ground-truth proofs for only 46 of 189
  • number: 189 standalone programs
  • limits: Finite tests miss missing constraints. It trusts the reference spec. Tasks are simple and standalone, with no interleavings, fairness or liveness. ‘Soundness/completeness’ here means containment between sets of programs, not soundness of the verifier.
  • how much was read: most of it read
  • source; notes: Generated contracts and independent evaluation; notes: VERINA; notes: papers
  • record id: verina-2026
• VeriSoftBench (2026)
  • title: VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
  • what: Introduces 500 Lean proof obligations drawn from real repositories, evaluating generation-check-repair agents under curated versus full repository context. The pipeline samples Pass@k proof scripts, compiles them with repository toolchains, and repairs failures using compiler errors and proof state. Dependency footprints and nesting depth are measured; large repositories exceeding 200k tokens expose context-window effects. Models include GPT-5.2, Claude Opus-4.5, Gemini-3-Pro, Goedel-Prover and Aristotle.
  • number: Gemini-3-Pro 41.0% curated/34.8% full; Claude 31.2%/23.2%; GPT-5.2 12.6%/10.8%
  • number: Aristotle 69% and Gemini 65% on 100-task compatibility subset
  • number: Large repositories: Gemini 17.6%/10.9%; depth 7+ 21.1%/15.6%
  • number: Average transitive dependencies: 11.2 library and 37.9 project; ~50% depth≥5
  • limits: Success requires compilation within 300 s and depends on repository toolchains, context packaging and repair budgets. Curated context uses ground-truth dependencies, while full context can overwhelm models. Results are Lean-specific and benchmark repositories may overlap model pretraining.
  • how much was read: most of it read
  • record id: verisoftbench-2026
• Vero (2026)
  • title: Vero: Can AI Agents Build Formally Verified Software Repositories?
  • what: 43 multi-module Lean repositories with supplied APIs and 2,705 specifications. Agents either write code plus proofs, or fill in proofs for given code.
  • number: Codex/GPT-5.5 xhigh: 27/43 joint, 25/43 proof-only
  • number: verified_ironkv: 19/22 joint, 22/22 proof-only
  • number: Audits find contradictory specs and proof/runtime mismatches
  • number: 43 instances
  • limits: The 22 IronKV targets are not the full deployed IronKV proof. implemented_by lets tests run one implementation while the theorem describes another.
  • how much was read: most of it read
  • source; notes: Vero: repository generation, including a limited IronKV instance
  • record id: vero-2026
• Verus issue 3188 (2026)
  • title: A malformed #![trigger] attribute on ensures deletes the ensures clause without an error
  • what: This open issue reports that a malformed trigger annotation can cause Verus to drop an ensures clause and still report successful verification. The reproducer compiles and runs a function returning 15 while the written postcondition says the result is 10.
  • number: Reproducer reports 1 verified and 0 errors
  • number: Concrete run prints compute(5) = 15 while the written verified postcondition is r == 10
  • limits: The report targets a rolling Verus revision and is not a corrected release. It shows parser/error-handling unsoundness, not a distributed-system case.
  • how much was read: most of it read
  • source
  • record id: verus-issue-3188-2026
• Verus-SpecGym (2026)
  • title: Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization
  • what: It judges agent-written Verus specifications by running them against official and adversarial Codeforces tests.
  • number: 581 tasks; best Gemini-3.1-Pro 77.8%; LLM judges miss 26% of detected failures
  • number: 581 specification-writing tasks
  • number: an LLM judge misses 26% of failures the evaluator catches
  • number: 75-minute timeout per run
  • limits: The tasks are algorithms. Testing is evidence, not a proof of equivalence in general.
  • how much was read: most of it read
  • source; notes: Dafny, C verification, and the specification-intent gap; notes: Generated contracts and independent evaluation; notes: Verus-SpecGym / Verus-SpecBench
  • record id: verus-specgym-2026
• Verus-TLA repository (2026)
  • title: verus-tla export (BasisResearch/verus PR #51)
  • what: This crate embeds TLA temporal predicates in Verus and provides state-machine structures, fairness predicates and verified helper lemmas for safety and liveness proofs.
  • number: 8 states
  • limits: README-level documentation; no independent proof audit was run.
  • how much was read: most of it read
  • source; notes: R02. Verus temporal libraries, 2025–2026 repository leads
  • record id: verus-tla-repository-2026
• Welder (2026)
  • title: Welder: Compositional Liveness Verification of Cluster Control Planes
  • what: Welder proves, for each controller, that it makes progress and limits its requests. This statement is called the CORE spec. It then combines the proofs of controllers that fit together. The authors verified the ReplicaSet, Deployment, StatefulSet and RabbitMQ controllers.
  • number: 1.5 person-years
  • number: proof lines: CORE 37,487 (ESR 36,215); executable 3,180; model 3,915; trusted 1,081; about seven minutes to check
  • number: analysis of 188 reported controller bugs claims CORE excludes 178 (95%)
  • number: seven bugs found during verification; zero in 629 functional, 1,302 controller-crash, 677 Pod-crash tests
  • limits: Features left out of the success condition are not checked. The proofs assume that faults eventually stop. Controllers that depend on each other in a circle need separate reasoning. Four controllers lack full feature parity with the upstream versions. The evidence on AI is qualitative only.
  • how much was read: most of it read
  • source; notes: Welder: Compositional Liveness Verification of Cluster Control Planes; notes: P07. Welder: Compositional Liveness Verification of Cluster Control Pl
  • record id: welder-2026
• Wirehead Rocq proofs (2026)
  • title: Toward Formal Verification of a Lock-Free Distributed Double Buffer Cache
  • what: A master’s thesis that proves, in the Rocq proof assistant, six properties of the swap algorithm in Wirehead, a MongoDB-based cache that lets many data generators feed one training job. The author models each MongoDB call as a plain function and the algorithm as a state machine where one process takes one atomic step at a time.
  • number: 4 Rocq files, 3,470 lines (thesis text says 3,475 in one place), 144 proofs, 2 Admitted, 3 axioms (all on the user-supplied chunk function)
  • number: Safety.v 1,339 lines; Liveness.v 1,342 lines; DB.v 513; Algorithm.v 276
  • number: about 800 lines of Safety.v are auxiliary invariants needed only for liveness
  • number: 5 of 6 properties fully checked; progress depends on 2 Admitted lemmas
  • limits: Assumes one MongoDB instance, crash-stop failures only, atomic database calls, and weak fairness of the scheduler; the match between the Rocq model and the Python code is asserted, not proved. Crash transitions and the real timeout-based lock recovery are not modeled.
  • how much was read: most of it read
  • source; notes: Toward Formal Verification of a Lock-Free Distributed Double Buffer Ca
  • record id: wirehead-rocq-proofs-2026
• Astrogator (2025)
  • title: Towards Formal Verification of LLM-Generated Code from Natural Language Prompts
  • what: Introduces Astrogator, a verifier for LLM-generated Ansible programs. A user-readable Formal Query Language gives a precise, reviewable specification of natural-language intent; a State Calculus models stateful system effects; a symbolic interpreter and unifier compare generated playbook behavior with the query and report extra assumptions/actions. The formalization separates user intent U from an approved but possibly under-specified query semantics and explicitly defines the unresolved gap p∖s for user review. The State Calculus models attributes, nested elements, stateful functions, reads, writes, conditionals, cases and symbolic loops; unknown reads branch over possibilities, and symbolic lists are approximated by one representative element. Ansible modules are specified in a small module-definition language, while a hand-curated knowledge base maps high-level descriptions such as apache server or zsh configuration file to OS-specific packages and paths. On 1,260 generated playbooks from six LLMs, Astrogator accepts most test-correct programs and rejects most incorrect ones, but under-specified queries permit incorrect programs unless users review reported assumptions.
  • number: 21 benchmarks
  • number: 1,260 generated programs
  • number: 334 correct (26.5%)
  • number: 277/334 accepted (82.9%)
  • limits: The formalizer cannot always infer intent from ambiguous natural language, so the user must approve the formal query. The query is intentionally incomplete and the evaluation accepts all programs satisfying it without user review of p∖s. The Ansible compiler does not support all features, especially shell commands, blocks and handlers; symbolic-loop handling is imprecise and arbitrary loops are future work. Ground truth for 1,260 programs is based on tests across Debian 12.11, Ubuntu 24.04.2 and RHEL 9.6 VMs, not a proof of real-world correctness. Knowledge-base single-valued choices cause false rejections when multiple implementations are valid.
  • how much was read: most of it read
  • source; notes: ## Astrogator, checking LLM Ansible code against a readable query (202
  • record id: astrogator-2025
• Atmosphere (2025)
  • title: Atmosphere: Practical Verified Kernels with Rust and Verus
  • what: Atmosphere is a verified kernel written in Rust and checked with Verus. The notes use it to count human effort and proof size.
  • number: proof-to-code ratio 3.32:1; replay under 20 seconds; under 2.5 person-years
  • number: 6K executable lines, 20.1K proof/spec lines: 14.3K spec and 5.8K proof/hints; 2.9K of specs for the top-level syscall model
  • number: 6K executable lines, 3.32:1 proof/code ratio, under twenty seconds verification, under 2.5 person-years
  • number: less than 2.5 person-years
  • limits: The source contradicts itself. Section 6.3 says about two person-years, with fourteen months on verified code. The abstract and introduction say under 2.5 person-years in total and 1.5 for verification. The guarantees cover the kernel’s functional correctness and isolation. They do not cover liveness of distributed systems.
  • how much was read: most of it read
  • source; notes: Atmosphere: count specification work and acknowledge inconsistent tota; notes: Atmosphere: Practical Verified Kernels with Rust and Verus
  • record id: atmosphere-2025
• Ax-Prover (2025)
  • title: Ax-Prover: a deep reasoning agentic framework for theorem proving in mathematics and quantum physics
  • what: Three LLM agents (orchestrator, prover, verifier) write Lean 4 proofs by calling Lean tools through the Model Context Protocol (a standard way for an LLM to call external tools). Tested on math and quantum-theory theorem sets, and in two expert-collaboration case studies.
  • number: NuminaMath-LEAN (300 sampled problems): 51% vs 28% DeepSeek-Prover-V2, 31% Kimina-Prover, 5% plain Claude Sonnet 4
  • number: AbstractAlgebra (100 problems, new): 64% vs 24% and 13%
  • number: QuantumTheorems (134 problems, new): 96% vs 61% and 57%
  • number: PutnamBench: 14% (92 of 660) with Claude Sonnet 4.5, third overall; top open-source entry
  • limits: Lean statements are given by humans (the new datasets were formalized by LLM plus expert review); all targets are math or physics theorems, not software or distributed systems. Pass@1 is defined as one multi-step agent run, so it is not the same budget as the baselines’ single shot.
  • how much was read: most of it read
  • source; notes: Ax-Prover (2025)
  • record id: ax-prover-2025
• b-invariant regular model checking (2025)
  • title: Regular Model Checking Upside-Down: An Invariant-Based Approach
  • what: Reverses regular model checking direction: intersects all inductive b-invariants represented by CNF formulas with at most b clauses, obtaining regular over-approximations converging to reachability. Defines regular transition systems with finite-word configurations and length-preserving transducers, then constructs automata for IndInv_b. For every b, unsafe-intersection emptiness is in EXPSPACE; b=1 has an exponential NFA and PSPACE-completeness. The proof uses tables/separation languages and a reduction from polynomial-space Turing machines. MONA experiments compute minimal DFAs for IndInv_1 across classic parameterized protocols and inspect cases needing b>1.
  • number: IndInv_1 proves 46/59 benchmark properties
  • number: minimal DFA and MONA times range <1 ms to 1920 ms
  • number: b=1 PSPACE-complete; every b EXPSPACE upper bound
  • number: dining philosophers with fork ownership needs b=1; unlabelled forks require b=3
  • limits: Only length-preserving finite-state transducer transitions and regular initial/unsafe sets are covered; safety problem for RTS is generally undecidable. EXPSPACE result is an upper bound and authors conjecture EXPSPACE-completeness for b≥2. IndInv_1 can be too coarse (e.g., some mutual exclusion/cache properties), while manual reasoning shows b=2 or b=3 can suffice.
  • how much was read: most of it read
  • source; notes: Regular Model Checking Upside-Down (2025)
  • record id: b-invariant-regular-model-checking-2025
• Basilisk (2025)
  • title: Basilisk: Using Provenance Invariants to Automate Proofs of Undecidable Protocols
  • what: A Dafny-based tool generates history-preserving asynchronous protocol models and mechanically checked provenance, monotonicity and ownership invariants. Atomic sharding traces current local values or collection entries to past steps and messages. Users supply host definitions, selected hints/labels and final safety proofs. Evaluation reports 16 protocol-model proofs, with 10 Kondo comparisons, final proof sizes and verifier times. Reconstructs host/network provenance and the atomic-shard algorithm, collection correlation failure and epoch-reset counterexample. Records formal safety obligations, all 16 protocol properties and every numerical comparison cell. Separates generated invariant proofs from user safety proofs, model correctness from implementation correspondence, and measured final LOC from development effort. Preserves incompleteness, syntax restrictions and all trust limits.
  • number: 16 protocols
  • number: Paxos: 418 lines, 0 user invariants, 486 proof lines, 24.6 s
  • number: Multi-Paxos: 447 lines, 0 user invariants, 565 proof lines, 61.5 s
  • number: Paxos row: 418 model lines, no user invariant clauses, five monotonicity annotations, no provenance hints, 444 safety-lemma lines, 486 total proof lines, 24.6 verification seconds
  • limits: §§3,5.2: Model has arbitrary host counts, nondeterministic single-host atomic steps, history of all Hosts snapshots and a monotonically growing set of sent messages. Receive selects an addressed message already sent; repeat selection and nondelivery overapproximate duplication, delay, reordering and loss. Messages carry sender identities; arbitrary Byzantine forgery or compromised hosts are not modeled. History is proof state, not a measured runtime logging cost. §3: Network provenance links a sent message to an adjacent pair of historical sender states and a matching send step. Host provenance for witness q requires q now, no initial state satisfying q, and a historical crossing from not-q to q through a candidate modifying step. These statements concern well-formed initialized protocol executions, not arbitrary malformed history sequences. §§3.3,5.3: Provenance alone identifies a past producing event. Monotonicity/non-equivocation invariants are needed to move from past sender conditions to present facts. Ownership invariants/labels are inherited from Kondo; generated invariants are individually/local mechanically checked, not assertions trusted merely because the generator produced them. §4.1: An atomic shard is a subset of local variables always modified together. If the current tuple is not an admissible initial tuple, it was established in one candidate update step. Maximal shards retain shared-message correlations lost by independent singleton witnesses. This does not mean every member of a collection remains correlated with the latest scalar value. §4.2: Static footprints overapproximate variables a step may update; excess candidates weaken the resulting provenance disjunction. Underestimating footprints would produce incorrect invariants. Prototype rejects a step updating different variables under different conditions, e.g. if c then x:=5 else y:=6. User must split this into separately guarded steps. §§4.2,5.3: Collection fields also receive refined singleton shards, allowing each noninitial set/map entry to be traced to its insertion step. Prototype supports explicit Dafny record-update syntax only, rather than general relational updates. A step can receive and send together; multiple sends/broadcasts in one step are described as an easy extension but not implemented or evaluated. §5.4 reader qualification: Formal obligations are Init⇒I∧φ and I∧φ∧Next⇒I′∧φ′. Thus the actual method proves safety with induction on I∧φ and may use current φ in the step proof. Figure 5/prose describing a proof that I implies safety must not be read as requiring or establishing the stronger standalone I⇒φ in every example. The final safety proof remains user-written. §§5.4,6.4: Failure may reflect a too-weak generated invariant or an unsafe model; the verifier points to the failing host-step obligation, but does not automatically decide which explanation is true. Reader qualification: SMT proof failure can also arise from solver incompleteness. Additional provenance witnesses or manually written invariant clauses may be needed; Basilisk is explicitly incomplete for undecidable logics. §6: First six protocols were used to develop/refine provenance and atomic sharding; the remaining ten were applied later. All sixteen models/invariants are reported outside EPR. This is an author-built benchmark suite, not a blinded independent user study or proof of completeness for all protocols outside EPR. §§6.1–6.2: Zero manually supplied invariant clauses across 16 does not mean zero user reasoning. Six of 64 host-provenance invariants need explicit witness hints, across Paxos-Dynamic, Raft election and Multi-Paxos; monotonicity annotations, ownership labels and safety lemmas remain. Distributed Lock and both ShardedKV variants already had fully automatic Kondo ownership proofs. §6.2: Kondo comparisons cover only ten protocols and modify models to split receive/send; Kondo also needs synchronous protocol definitions. Final proof LOC excludes failed/revised proof attempts and does not measure person-time, cognitive difficulty or novice success. Basilisk uses more proof LOC for some simple cases and is not faster for every ownership example. §6.3: Timings check final history-preserving asynchronous proofs including generated/user invariants and lemmas; Kondo synchronous-proof verification is excluded. Invariant generation is under two seconds in every case. Table 1 reports Multi-Paxos 61.5 seconds; the prose phrase in a minute is approximate, not a strict sub-60-second bound. No repetitions/variance or controlled proof-authoring experiment is reported. §6.4: Epoch resets create implicit multi-step relations that sharding cannot infer. Adding an event at current year and clearing events on year increment requires a witness relating every current entry to current year; merely finding its historic insertion year is too weak. All three hinted benchmark protocols have collection resets at epoch changes. Footprint imprecision is another possible weakness, not observed by authors in these protocols. §6.4: Safety specification is trusted. The user must trust the model describes actual code or separately prove correspondence; this work does not verify production implementations. Dafny, dependencies and hardware are trusted. Crash-fault safety is the scope; no liveness or Byzantine correctness proof is provided. §§6–8,Appendix A: No runtime throughput, production bug-finding or LLM-agent experiment is evaluated. Artifact developed/tested on M3 MacBook Pro with MacOS Sequoia; this appendix statement is not an explicit platform declaration for every published timing. Artifact and external proof bodies were not executed in this reading.
  • how much was read: most of it read
  • source
  • record id: basilisk-2025
• CCF smart casual verification (2025)
  • title: Smart Casual Verification of the Confidential Consortium Framework (CCF)
  • what: Industrial experience applying TLA+ specification/model checking, weighted simulation and implementation trace validation to the existing C++ Confidential Consortium Framework. Separate node-consensus and client-history specifications are aligned with production protocol details and incorporated in CI. The paper reports six consensus bugs found/fixed and a clarified read-only consistency limitation, plus tooling/effort and state-exploration measurements. Reconstructs all six safety/liveness cases and the unsuccessful first fix, distinguishes trace compatibility from universal implementation refinement, records granularity alignment via composition/stuttering and incomplete-state reconstruction. Saves all scale/coverage rows, effort figures, counterexample sizes and issue pointers. Preserves TEE/model trust, bounded exploration and non-linearizable read-only semantics; no all-executions C++ correctness theorem is claimed.
  • number: Six subtle bugs found, five concerning safety and one liveness
  • number: 400 LoC Trace spec; 88 commits to it and 107 changes to the consensus spec; implementation 63 kLoC C++
  • number: six subtle bugs found
  • number: 63 kLoC in C++ (§2)
  • limits: §§1–2: CCF runs application logic inside trusted execution environments while treating operators, host OS/hypervisor, network and storage as corruptible. This paper checks distributed protocol behavior under modeled failures, not hardware/firmware isolation, enclave security, cryptography, all 63k C++ LOC, or arbitrary Byzantine behavior of the protected consensus code. Production use in Azure Confidential Ledger is context, not evidence that every production behavior is covered. §§1.1,4,8: Smart casual verification combines formal specification, finite model checking/simulation and tests/trace checks. Consensus spec is unbounded but exhaustive checking restricts terms, requests and reconfiguration sequences. No TLAPS safety/liveness proof for all sizes or universal code-refinement proof is reported. Most CI exploration relies on bounded simulations because sufficiently realistic exhaustive exploration is too expensive. §2: CCF returns transaction results before replication; clients separately query Pending/Committed/Invalid status. A returned result may later be invalidated. Committed read-write transactions are strictly serializable; committed read-only transactions are serializable and may be stale. Pending branches offer fork-like consistency rather than arbitrary results. Broad prose about all committed transactions being linearizable needs the explicit read-only exception. §2: Transaction IDs are lexicographic term/index pairs. If two transactions commit, ID ordering agrees with execution ordering. Ancestor Commit says committed <t,i> implies same-term <t,j>, j≤i, committed. This concerns transaction identity/status, not a proof that any stale read immediately observes the newest committed write. §2.1: CCF customizes Raft with signed Merkle-root transactions, messaging without RPC correspondence, optimistic sentIndex advances, express catch-up, CheckQuorum, bootstrapping/reconfiguration and retirement/ProposeVote. A transaction is committed only after a subsequent signature is committed. The actual combined protocol needs verification; citing the base Raft proof does not establish these extensions. §2.1 reader qualification: Early description says an AE-ACK carries the last transaction in the follower log, while the Inaccurate AE-ACK bug/fix later constrains that index to the end of the received AE. The historical/protocol-description distinction must be retained rather than treating the early sentence as the final safe response rule. §§2.1,4: Reconfiguration needs majorities in each active configuration, rather than a majority of their union. A removed node remains retiring until retirement information is safely committed for future leaders. CheckQuorum steps down an isolated leader, but absent delivery/scheduling assumptions this is not a theorem that all executions make progress. §4: Consensus has 17 actions/13 variables, 12 local-state mappings plus network. Parameterization includes every nonempty initial node subset with any member as initial leader. Locally initiated timeout/signature/candidacy/step-down choices impose no clock synchrony. LogInv compares pairwise committed-log prefixes; AppendOnlyProp prevents committed-prefix rollback over time. Another 27 properties include MonoLogInv, requiring term increases only after signatures. §4: Simulation explores behaviors to bounded depth under a time quota; authors manually downweight failures to promote forward progress. Q-learning was implemented in TLC but no state-hash feature set beat manual weighting. More progress-rich traces may help find bugs but do not constitute exhaustive failure coverage or proof of liveness. §5: Client model uses append-only history and logBranches, independent of node count. History includes read-only/read-write requests/responses and status responses, but omits status requests/Pending replies. A simple shared value appends transaction identifiers so conflicts and observations are visible. It is a deliberately selected application workload, not a proved reduction of all arbitrary CCF application semantics. §5: A new branch can be any prefix of an existing branch containing the last committed transaction. Requests may execute at any still-active leader/term. PrevCommittedInv checks same-term committed ancestor statuses; ObservedRoInv demands that a subsequently committed read-only response observe a read-write transaction whose response preceded its request. The latter expectation is falsified and documented, not a proved CCF invariant. §6.1: Scenario driver serializes consensus execution deterministically and mocks governance/networking/other components, injecting loss/reordering/partition/delay. Original 13 scenarios and about 1.6k unit-test/2k Python end-to-end LOC precede the effort; Table 1 gives later exact scope counts. Initial fuzzing was abandoned for insufficient interesting coverage. Driver/mock accuracy is a boundary, as demonstrated by retirement bugs. §6.1: Driver uses a global logical ordering clock; distributed Lamport/vector alternatives are suggested but not implemented here. Fifteen added logging statements capture stable states at message and role-change points, immediately after a global lock for state changes. Constant-space values such as log lengths are recorded, not full log entries. Production logging compiles out, so the checks use instrumented test executions rather than continuous production monitoring. §6.1/§6.5: Traces initially remove/deduplicate bootstrapping events because early consensus spec omits that phase; later specification is expanded to model bootstrapping more faithfully. These describe successive coverage stages. A successful preprocessed trace validates only its retained observations and modeling assumptions, not excluded initialization behavior. §6.2: Trace spec checks existence of a spec behavior consistent with observed events/values: T∩S is nonempty. Nondeterministically reconstructed hidden state is existential. A successful finite trace check is evidence for those observations, not proof that all completions, all execution traces or unlogged state transitions meet the spec. If logging or mappings omit a decisive distinction, an incorrect implementation could remain compatible. §6.2: Send actions constrain current state, invoke high-level action and assert matching successor messages. Network changes from set to multiset to represent duplicate/resend occurrences. Trace network abstractions also explore ordering/duplication/loss guarantees. Such abstraction choices affect what is observable and accepted; they are not runtime network guarantees inferred from traces. §6.2.1 Listing 5: UpdateTerm·HandleAppendEntriesReq composes two model actions into one implementation receive. IsFault·Next admits unlogged network faults before a step; send-response events may stutter high-level variables when already modeled by another action. Composition hides intermediate states and stuttering changes observation granularity, so correctness depends on the mapping retaining semantically important events. Merely making a mismatch disappear is not an independent proof. §6.2.2–6.3: Start with a happy-path trace, add many assertions and cross-references, debug source and formulas. Invalid traces may indicate bogus logging, incorrect mapping, specification error or implementation bug; failure lacks an ordinary model-checking counterexample. Longest matching prefixes, unreachable-state graphs, failing subformulas and an unsatisfied breakpoint support diagnosis. §6.4: DFS stops after finding one compatible behavior, avoiding BFS enumeration of alternatives; a consistency trace goes from about one hour to under a second. This is an author example without trace-size distribution or controlled speedup suite, and does not mean all traces validate in under a second. Existential acceptance is essential to that optimization. §6.5: Logging/driver enhancements took about one day. Consensus alignment/tooling took two engineer-months over four months, including TLC composition/DFS/debug support, discrepancy diagnosis and abstraction design. Writing about 400 lines of Trace was minor; Table 1 counts 369. Trace saw 88 commits and consensus spec 107 changes. This is observed project effort, not a controlled productivity study. §6.5: Client consistency validation uses REST observations without source instrumentation, reconstructs other clients’ hidden transactions from observed IDs, and took about one engineer-week over two weeks, mainly by CCF experts. Smaller spec, previous experience and reused TLC changes confound a simple comparison with initial consensus effort. §7 Table 1: Approximate state counts/rates are measured on Azure DC8s v3. A log line is largely equivalent to a model action, according to authors, not a formal equality of semantic coverage. Different abstraction sizes, bounds, durations and distinct/generated-state counting prevent treating orders-of-magnitude state counts as a calibrated defect-detection probability. §7: Six serious consensus bugs comprise five safety issues and one liveness issue, found by combined modeling, source scrutiny, trace mismatch, simulation and functional/end-to-end reproduction. They are not six independent discoveries by model checking alone. Read-only non-linearizability is an additional documented consistency ambiguity and accepted behavior, not a seventh fixed consensus bug. §§7–8: Authors report all six fixed before user impact/customer reports; software had already been deployed. One-node-at-a-time operational reconfiguration masked the election bug, and absence of an incident from the first bad commit fix was attributed to chance. No external incident audit, current repository verification or independent bug reproduction was done in this reading. §§8–9: Paper chooses model checking for developer accessibility rather than deductive proof, despite compute cost. AI-assisted proof search is future possibility with a reference to neural theorem proving, not an experiment using LLM agents. Related-work sentence says IronFleet and Verdi both proved Raft; reader qualification: IronFleet uses a Paxos-based replicated state machine, so do not propagate that comparison as fact.
  • how much was read: most of it read
  • source
  • record id: ccf-smart-casual-verification-2025
• CLEVER (2025)
  • title: CLEVER: A Curated Benchmark for Formally Verified Code Generation
  • what: CLEVER is a Lean benchmark. A model generates a spec, which must be proved equivalent to a hidden, human-written reference spec. The model then generates code, which must be proved against that reference. The reference specs are noncomputable so that models cannot copy code logic.
  • number: 161 problems (from 164 HumanEval; 3 excluded)
  • number: default budget 600 seconds for pass@k-seconds
  • number: human effort per problem: 25 minutes plus an additional 15 minutes, in places over an hour
  • number: reference proofs 10-225 lines
  • limits: It depends on the quality of the reference. Failed equivalence proofs give false negatives. HumanEval data may have leaked into training. It covers only list and numeric tasks, and has no liveness.
  • how much was read: most of it read
  • source; notes: Generated contracts and independent evaluation; notes: CLEVER
  • record id: clever-2025
• Crossword (2025)
  • title: Crossword: Adaptive Consensus for Dynamic Data-Heavy Workloads
  • what: # Crossword: Adaptive Consensus for Dynamic Data-Heavy Workloads (authored by agents unless marked 🧑) Guanzhou Hu, Yiwei Chen, Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau, arXiv:2509.07157v1, 8 September 2025. This reading covers all 30 PDF pages: design, implementation, every evaluation, discussion, 144 references, complete PlusCal/TLA+ specification and model-checking configuration. All eight contact sheets and close views of pages 5, 8, 10–12 and 19–30 were read. PDF and printed page numbers agree. The findings below distinguish author reports from static reader reasoning; no author code or checker was run. [Evidence: Q0, title, abstract and §1, PDF pp.1–2; Q24, Appendix A model-check report and explicit finite scope, PDF pp.19–19; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.] Crossword changes how much erasure-coded data each replica must durably store before a log entry commits. More data per replica permits a smaller reply quorum; less data requires more replies. Its motivating Figure 1 profiles Raft payloads under 200-warehouse TPC-C in TiDB and CockroachDB: “~45% are 4KB” and the distributions reach 290 KB and 63 MB, respectively. The paper derives a shard-survival condition, combines it with majority voting, and implements per-entry adaptation in Rust and in a CockroachDB Go prototype. Followers exchange missing shards after commitment. Its evidence is algebraic design reasoning, a reported finite model check, tests and system measurements. The paper says, “It has been model-checked for both linearizability and fault-tolerance properties” and immediately names the finite inputs. The artifact supplies neither an unbounded proof nor a refinement from the model to Rust or Go. [Evidence: Q4, §3.3 complete derivation and Figure 4, PDF pp.5–5; Q7, §3.6 complete protocol changes, PDF pp.7–7; Q8, §4 implementation and Gazette/CockroachDB, PDF pp.7–7; Q24, Appendix A model-check report and explicit finite scope, PDF pp.19–19; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.] ## Assumptions and the actual contract The system model uses an odd number n of replicas, asynchronous communication and fail-stop failures. Set m=ceil(n/2) and f=n−m. Balanced policies use Reed–Solomon (n,m) coding, so any m distinct shards reconstruct a payload. A vote means both acceptance of a ballot and durable storage of its assigned shards: “s votes yes to the ballot number of this Accept message and has durably remembered the shards it carried.” The leader retains the entire payload in memory but persists only its assignment. Background gossip remains in memory. Restart recovery, Byzantine behavior and cryptographic shard authenticity are outside this model. Time-based read leases separately assume bounded clock drift; the asynchronous network model alone does not establish lease correctness. [Evidence: Q1, §2 complete background, system model and motivation, PDF pp.2–3; Q3, §3.1–3.2 assignments and Figure 3, PDF pp.4–5; Q4, §3.3 complete derivation and Figure 4, PDF pp.5–5; Q6, §3.5 gossip and Figure 5, PDF pp.6–7; Q11, §4.3 heartbeats, snapshots and leases, PDF pp.8–8; literal spans in quotes.json.] For an acceptance pattern ap, C1 requires at least m distinct replying nodes, including the leader. C2 requires at least d distinct recoverable shards after removing any f replying nodes, with d=m for the balanced family. Majority intersection preserves the voting argument; the second condition preserves payload availability after the allowed crashes. These are different obligations. C2 alone does not prove preservation of the chosen value across changing ballots and assignments, and neither condition promises termination in an arbitrary asynchronous execution. [Evidence: Q4, §3.3 complete derivation and Figure 4, PDF pp.5–5; Q7, §3.6 complete protocol changes, PDF pp.7–7; Q28, Appendix A PreparedConditionAndCommand, PDF pp.22–23; literal spans in quotes.json.] Balanced round-robin policies assign c consecutive shard indices to each node, wrapping around the n indices, with 1≤c≤m. The derived valid region is n≥q≥m and q+c≥n+1; the candidate performance boundary is q+c=n+1. For n=5,m=3,f=2, its three choices are (c,q)=(1,5),(2,4),(3,3). Full replication and single-shard coding become endpoints of one family. Section 3.2’s written range s through s+c would contain c+1 shards if both endpoints were included; the figures and Appendix A’s upper endpoint ShardToIdx[r]+na−1 establish the intended c-shard interpretation. Figure 2 gives the motivating failure: with single shards and q=4, crashing the leader and another accepting node leaves only two shards, below the three needed. The paper’s RSPaxos/CRaft comparisons commonly use f=1, whereas Crossword retains f=2. Their throughput comparisons therefore include different availability guarantees, which the graphs label. [Evidence: Q2, §2.4 failure example, Figure 2, PDF pp.3–3; Q3, §3.1–3.2 assignments and Figure 3, PDF pp.4–5; Q4, §3.3 complete derivation and Figure 4, PDF pp.5–5; Q5, §3.4 tradeoff, PDF pp.6–6; literal spans in quotes.json.] One printed intermediate bound needs a saturation qualification. The paper writes SubCover(ap,f)≥q−f+c−1, but shard coverage cannot exceed n. Reader example: n=5,q=5,c=3,f=0 gives a right side of seven although only five distinct shards exist. The round-robin lower bound should be capped by n. The later threshold test against m≤n, including q+c≥n+1 at f=n−m, is unaffected. This is arithmetic about the displayed derivation, not an implementation finding. [Evidence: Q4, §3.3 complete derivation and Figure 4, PDF pp.5–5; literal spans in quotes.json.] ## Protocol, adaptation and concrete cases Each Accept message carries the chosen per-instance assignment; recipients persist their assigned bytes before acknowledging. On failover, a majority Prepare exchange preserves a reconstructible highest-ballot value; the paper permits replacing it with a new command if fewer than d shards of that value can be recovered. The appendix selects a maximum-ballot command and combines reported shards for that command, including matches at other ballots. A complete correctness argument must connect that rule to chosen-value preservation over all earlier ballots, not just count surviving shards in one acceptance pattern. This reading identifies that proof obligation without claiming a counterexample. [Evidence: Q3, §3.1–3.2 assignments and Figure 3, PDF pp.4–5; Q7, §3.6 complete protocol changes, PDF pp.7–7; Q28, Appendix A PreparedConditionAndCommand, PDF pp.22–23; literal spans in quotes.json.] Figure 5 distinguishes committed-but-partial data from committed-and-known data and execution. Followers periodically request missing shards from peers; reconstructed shards are memory-only. Requests are batched about every 20 ms and normally exclude the last 400 KB of accumulated payloads. A peer without a reconstruction reply for ten cycles is skipped. A new leader reconstructs its partial tail synchronously. Low-traffic tails, retrying skipped peers, eventual delivery and leader recovery consequently matter to progress; the text provides mechanisms and experiments rather than a temporal proof covering them. [Evidence: Q6, §3.5 gossip and Figure 5, PDF pp.6–7; Q10, §4.2 gossip details and Figure 6, PDF pp.8–8; Q11, §4.3 heartbeats, snapshots and leases, PDF pp.8–8; literal spans in quotes.json.] The default chooser fits per-follower response time t_s(v)=d_s+v/b_s from a two-second sliding window, updates every 200 ms, and discards the slowest five percent of observations. It considers balanced assignments and uses the (q−1)th smallest follower response-time estimate, because the leader already counts toward q. The stated limitations are no automated unbalanced search and no reaction to fluctuations shorter than roughly one second. Heartbeats and random 300–600 ms election timeouts track failures; the healthy-node count limits q. The CockroachDB integration instead uses payload thresholds, not this regression chooser. [Evidence: Q8, §4 implementation and Gazette/CockroachDB, PDF pp.7–7; Q9, §4.1 complete regression chooser and limitations, PDF pp.7–8; Q11, §4.3 heartbeats, snapshots and leases, PDF pp.8–8; literal spans in quotes.json.] A concrete source inconsistency occurs in §4.1, PDF p.8: the chooser evaluates t_s(v_p/c). Each of c assigned shards actually has v_p/m bytes, so the bytes per recipient are c·v_p/m. With n=5,m=3, c=1 sends v_p/3 but the printed expression predicts v_p; c=3 sends v_p but predicts v_p/3. Thus the printed expression reverses the size trend. It should not be used as the specification of an implementation without resolving the mismatch. No repository revision was inspected, so this is not a claim about executed code or the validity of measured results. [Evidence: Q3, §3.1–3.2 assignments and Figure 3, PDF pp.4–5; Q9, §4.1 complete regression chooser and limitations, PDF pp.7–8; literal spans in quotes.json.] ## Implementations and measured evidence Gazette is async Rust/Tokio with channels. The paper lists 12.7k infrastructure lines and six implementations: Chain Replication 1.1k, MultiPaxos 2.3k, Raft 2.3k, RSPaxos 2.5k, CRaft 2.6k and Crossword 3.4k. It reports “extensive unit tests and fuzz tests” but gives no counts, fault schedules or coverage. The Go prototype modifies CockroachDB v24.3.0a by approximately 1.6k lines. Those listed amounts total 26.9k lines, slightly below §1’s “a sum of 27.3k lines of async Rust”; the paper does not reconcile the counting scope. The paper promises to open-source the code; this reading establishes no repository version or reproduction. [Evidence: Q8, §4 implementation and Gazette/CockroachDB, PDF pp.7–7; literal spans in quotes.json.] Gazette experiments use CloudLab c220g2 hosts, 40 cores, 160 GB RAM and Linux 6.1.64. The regional setting has 1 Gbps links and 4 ms average delay; the wide-area setting spans CloudLab datacenters with about 200 Mbps and 30 ms average delay. Clients share hosts with servers but use disjoint pinned cores; batching is one millisecond. Default static tests have five servers and fifteen closed-loop clients. Requests are half lease-local Gets and half Puts, with eight-byte keys and small, 128 KB or mixed payloads whose normal distributions have ten percent standard deviation. Reported throughput includes reads; latency curves show means and arrows mark P95, not confidence intervals. [Evidence: Q12, §5 evaluation questions and §5.1 setup, PDF pp.8–9; Q13, §5.1 Figure 7 complete static comparisons, PDF pp.9–9; literal spans in quotes.json.] Figures 7–9 report that Crossword follows the favorable endpoint across small, large and mixed data: up to 1.9× coded-protocol throughput for small data, 2× MultiPaxos for large data, and 2.1× alternatives in the mixed workload. Figure 8 varies payload means from 8 B to 256 KB, replica counts 3/5/7/9 and write shares 10/50/100%; at ≥64 KB, the chosen c=1,q=5 case exceeds MultiPaxos throughput by four times in the plotted setting. Figure 8’s cluster-size/Put-share sensitivity reports “up to 2.3x over MultiPaxos” and 1.4× over RSPaxos in its tested settings. These are workload-specific ratios, not universal bounds. At 64 KB Figure 9 reports 71% less critical-path data-transfer/WAL time and logs of 913 MB versus 1,467 MB, a reader-calculated 37.76% storage reduction. [Evidence: Q13, §5.1 Figure 7 complete static comparisons, PDF pp.9–9; Q14, §5.1–5.2 Figures 8–9 and storage/critical-path measurements, PDF pp.9–10; literal spans in quotes.json.] Figure 10 changes conditions about fifteen seconds apart: payload size 64→4 KB, bandwidth 1 Gbps→100 Mbps→1 Gbps, and then two replicas with tenfold worse delay and bandwidth. Throughput is sampled once per second. Adaptation is shown by time traces, not a regret or convergence guarantee. Figure 11 crashes leaders at 15 and 55 seconds, uses a special 1.5-second timeout and disables snapshots. Crossword’s first recovery gap is about twice classic protocols’ gap because of tail reconstruction; afterward it selects c=2,q=4. RSPaxos becomes unavailable after the second crash, whereas CRaft completes its fallback before it. These particular separated failures do not test every crash spacing. [Evidence: Q15, §5.2 Figure 10 adaptation measurements, PDF pp.10–10; Q16, §5.3 Figure 11 crash measurements, PDF pp.10–11; literal spans in quotes.json.] Figure 12 manually chooses an unbalanced assignment for follower links of 1/1/0.4/0.1 Gbps, using the (8,5) codeword example with 5/5/3/1 assigned shards. It obtains roughly 1.4× MultiPaxos or balanced Crossword throughput. Automated unbalanced adaptation remains future work. Figure 13 forces c=1 and compares 0/400/800 KB deferral gaps at 3/10/100 ms write intervals, measuring follower version distance from the leader. Section 5.5 says, “The replies are not linearizable but sequentially consistent,” followed by “possibly stale versions in the history”. This is evidence about staleness, not a history-based proof of the text’s sequential-consistency claim. Figure 14, at fixed throughput, reports 13% less follower-to-follower bandwidth from batching; it moves two-thirds of payload replication to that background path. The 13% is not a reduction of all network traffic. [Evidence: Q17, §5.4 Figure 12 unbalanced assignment, PDF pp.10–11; Q18, §5.5 Figures 13–14 staleness and gossip traffic, PDF pp.10–11; literal spans in quotes.json.] Figure 15 uses YCSB-A key popularity on 1,000 keys, but all requests are Puts with a TiDB-derived size distribution. Five partitions rotate leaders over the same five regional servers. Crossword’s peak throughput exceeds MultiPaxos but falls below RSPaxos and CRaft, which omit its background gossip. Chain Replication has the highest unsaturated latency from chain propagation and does not reach its throughput limit over the plotted load range. This experiment tests shared follower bandwidth, not the usual half-read YCSB-A operation mix. [Evidence: Q19, §5.6 Figure 15 partitioned-store experiment, PDF pp.11–12; literal spans in quotes.json.] The CockroachDB TPC-C experiment uses twelve nodes, 200 warehouses, five replicas per range and 400 workers. It disables leader/leaseholder rebalancing and write pipelining. The chooser uses two shards above 4 KB and one above 8 KB. Aggregate throughput is 2,520 tps for vanilla, 3,330 for Crossword and 2,450 for an always-one-shard variant; reader arithmetic gives 32.14% gain over vanilla. The reported “up to 44%” refers to transaction-specific speedup. Table 1 separately measures RS(5,3) coding on 32 cores with SIMD: 4/16/64/256 KB and 1/4 MB take 1/4/16/77 μs and 1/6 ms; reported per-request leader CPU overhead is about 1.24–1.26%, with reported memory 1.6 KB through 1.6 MB. These costs have a different measurement denominator from throughput ratios. Repetition counts, warm-up and confidence intervals are not reported; the dynamic plots show their trace duration. [Evidence: Q20, §5.7 Figure 16 CockroachDB experiment, PDF pp.11–12; Q21, §5.8 Table 1 coding overhead, PDF pp.11–12; literal spans in quotes.json.] ## What the formal appendix actually checks Appendix A reports three replicas, at most one failure, two ballots, three requests and both balanced choices over three shards with two data shards. The supplied configuration specifically has three writes, no reads, CommitNoticeOn=FALSE and NodeFailuresOn, replica/write symmetry, TypeOK and Linearizability invariants, and deadlock checking. It gives no TLC version, explored-state counts, timing or run log. “These inputs should be large enough to explore all the distinct, interesting execution paths of the protocol” is the authors’ belief; no parameterized theorem justifies it. In particular, three nodes cannot instantiate the interior five-node policy c=2,q=4 or the unbalanced example. [Evidence: Q24, Appendix A model-check report and explicit finite scope, PDF pp.19–19; Q27, Appendix A coverage and valid round-robin assignments, PDF pp.21–22; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.] The PlusCal model represents shard identifiers, abstract command values and votes. It does not encode Reed–Solomon byte algebra, disk operations, the Rust or Go program, gossip/reconstruction messages, the chooser, leases, snapshots or reconfiguration. The single rloop label makes the macros atomic transitions; messages accumulate in a set. Nodes fail permanently. ReplicaCrashes additionally requires balMaxKnown<MaxBallot, suppressing crashes after learning the maximum modeled ballot: “∧node[r].balMaxKnown < MaxBallot” (Appendix A, PDF p.28). This is an explicit state-space restriction, not the protocol’s ordinary fail-stop rule. [Evidence: Q25, Appendix A constants, assumptions and state types, PDF pp.20–21; Q27, Appendix A coverage and valid round-robin assignments, PDF pp.21–22; Q29, Appendix A message types and commitment condition, PDF pp.22–24; Q35, Appendix A remaining process and MC definitions, PDF pp.29–29; Q34, Appendix A crash guard and process loop, PDF pp.28–29; literal spans in quotes.json.] The model’s Linearizability predicate is conditional: “terminated ⇒” a complete order exists that respects observed real-time precedence and old values. Terminated means no pending requests and all requests acknowledged. Therefore the checked expression constrains fully completed runs; arbitrary incomplete finite histories are not checked by that expression. Writes return their old value, so the all-write model still has meaningful observable ordering constraints, but it does not exercise read operations or leased local reads. Request identities are unique, admission picks the first unseen pending command, and InitPending fixes writes before reads under a stated W.L.O.G. reduction. Its comment says, “all other cases are either symmetric or less useful” (Appendix A, PDF p.21). No theorem establishes that reduction for a larger mixed workload. [Evidence: Q26, Appendix A initial request order and server states, PDF pp.21–21; Q30, Appendix A main state, termination and observation helpers, PDF pp.24–24; Q32, Appendix A complete new-request and Accept handling, PDF pp.26–27; Q36, Appendix A complete observable-order predicate, PDF pp.30–30; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.] No fairness condition or eventual-completion temporal property is supplied. CHECK_DEADLOCK tests the absence of an enabled next step under TLC’s deadlock semantics; it is not a proof that every admissible execution eventually completes. With CommitNoticeOn=FALSE, the loop can select a branch whose conditional body does nothing while a node remains live, illustrating why a no-progress execution need not be a deadlock. The displayed theorem statements have no TLAPS proof. Thus the author’s “fault-tolerance” report must be read together with the configured invariant, finite crash restriction and deadlock check, rather than promoted to unbounded liveness. [Evidence: Q34, Appendix A crash guard and process loop, PDF pp.28–29; Q35, Appendix A remaining process and MC definitions, PDF pp.29–29; Q36, Appendix A complete observable-order predicate, PDF pp.30–30; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.] Two further model-to-design obligations are concrete. TakeNewRequest puts all Shards in the leader’s voted.shards while its own AcceptReply records only assign[r]; the prose persists only assigned shards but expressly keeps the complete codeword in leader memory. Since failures are fail-stop with no restart, the abstract full-shard knowledge is consistent with a live leader’s memory. A refinement must explain that knowledge and its lifetime; this difference does not establish a crash-recovery durability counterexample. CommitNotice has only an index, no ballot or origin, and its handler applies a local command without shard reconstruction; the supplied configuration disables that handler. These are source differences and missing coverage, not checked exploit traces. The prepared-value fallback also spells a field “shard” rather than “shards”; HandlePrepareReplies awaits exam[s].prepared for every slot before reading shards, so that false-prepared branch alone does not establish a missing-field failure. No translator was run. The handler also overwrites cmd/shards while preserving a Committed status and sends its own reply before its voted record is updated by any later self-Accept. Chosen-value preservation and consistent self-voting must justify those intermediate states; no reachable counterexample is established. [Evidence: Q3, §3.1–3.2 assignments and Figure 3, PDF pp.4–5; Q28, Appendix A PreparedConditionAndCommand, PDF pp.22–23; Q29, Appendix A message types and commitment condition, PDF pp.22–24; Q32, Appendix A complete new-request and Accept handling, PDF pp.26–27; Q33, Appendix A commit and CommitNotice handlers, PDF pp.27–28; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.] ## Contribution to the human’s topic For distributed verification, this is a useful case separating voting intersection from durable data reconstructibility while policies change per instance. A proof would need cross-ballot chosen-value preservation, shard-survival and data-known/executed invariants, plus explicit assumptions for eventual gossip and failover. TLA+ can express the distributed contracts; Verus could connect assignment bitmaps, persistence-before-ack and reconstruction to Rust execution. Those are reader-proposed next steps, not supplied proofs. The static chooser mismatch gives a precise specification-to-code question. [Evidence: Q4, §3.3 complete derivation and Figure 4, PDF pp.5–5; Q7, §3.6 complete protocol changes, PDF pp.7–7; Q9, §4.1 complete regression chooser and limitations, PDF pp.7–8; Q8, §4 implementation and Gazette/CockroachDB, PDF pp.7–7; literal spans in quotes.json.] For evaluating LLM-generated specifications or proofs, the artifact offers concrete scope checks: whether an agent notices the all-write/CommitNoticeOn=FALSE configuration, completed-run predicate, ballot-dependent crash restriction, atomic actions, idealized shards and leader-knowledge abstraction. A convincing generated proof would have to state which model and which property it proves and show the link to implementation. The paper makes no claim of LLM-generated specifications, proofs or evaluation, and references Verus only as related verification work. Its strongest contribution here is a nontrivial Rust systems case with a published finite specification and measurable adaptation, together with explicit gaps a later proof effort must close. [Evidence: Q22, §6 discussion and §7 conclusion, PDF pp.12–13; Q24, Appendix A model-check report and explicit finite scope, PDF pp.19–19; Q32, Appendix A complete new-request and Accept handling, PDF pp.26–27; Q34, Appendix A crash guard and process loop, PDF pp.28–29; Q36, Appendix A complete observable-order predicate, PDF pp.30–30; Q37, Appendix A complete Crossword MC.cfg, PDF pp.30–30; literal spans in quotes.json.]
  • limits: Odd n, fail-stop, asynchronous network; majority m=ceil(n/2), f=n−m; balanced RS(n,m), c consecutive wrapped shards, q+c≥n+1. Durable assigned shards before ack. Bounded clock drift separately assumed for read leases. Printed chooser uses inverse shard count instead of assigned bytes; intermediate coverage bound needs saturation; cross-ballot preservation proof obligation; finite formal scope/reductions/crash restriction/completed-run predicate; leader-knowledge abstraction requires refinement; disabled commit notices; no Rust refinement; heterogeneous policies manual; <1s fluctuations outside chooser scope; f1 baselines differ; staleness not a consistency proof. Prose RR interval endpoint differs from intended c-shard range; listed line counts total26.9k versus headline27.3k; prepared-state overwrite/self-vote obligations remain conditional static observations.
  • how much was read: most of it read
  • source
  • record id: crossword-2025
• DafnyBench (2025)
  • title: DafnyBench: A Benchmark for Formal Software Verification
  • what: Asks LLMs to restore missing assertions and loop invariants in Dafny programs that already have specs. The pass rate measures whether the proof gets finished. It does not measure whether the intent was formalized.
  • number: 782 Dafny programs
  • number: 782 verified ground-truth files; over 750 programs and about 53,000 lines (abstract)
  • number: about 68% for the best model/prompt scheme
  • number: built with Dafny 4.3.0
  • limits: Only bans {:verify false} and assume false. This does not show that every way to cheat is caught. The public GitHub source may be in training data. Some specs just repeat the implementation. Scoring generated specs is only proposed (Appendix C). Verifying a weak contract that was handed in counts as success.
  • how much was read: most of it read
  • source; notes: DafnyBench
  • record id: dafnybench-2025
• DafnyComp (2025)
  • title: Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verification
  • what: The benchmark has 300 Dafny programs written by LLMs. Each has 2 to 5 chained functions, with contracts removed. The model must rebuild the contracts that connect the functions.
  • number: 300 programs selected from 564 verified of 1,200 attempts
  • number: 300 programs
  • number: 95.67% syntax correctness
  • number: 3.69% verification rate
  • limits: The review notes found no check against assume or admit (unproved assumptions) or against vacuous specs (specs that hold trivially). Over-restrictive requires clauses (preconditions) could make verification easier. The prompt in Appendix G contradicts Section 3.3. It excludes liveness, resource bounds and security policies. It has no concurrency.
  • how much was read: most of it read
  • source; notes: DafnyComp
  • record id: dafnycomp-2025
• Ellsberg (2025)
  • title: Runtime Protocol Refinement Checking for Distributed Protocol Implementations
  • what: It monitors the messages going into and out of a deployed process. It searches for an allowed run of a TLA+ protocol, translated to Go, that explains them. If none exists, it reports an error. It was applied to Etcd, ZooKeeper and Redis Raft.
  • number: executable spec sizes: ETCD 952, RedisRaft 894, Zookeeper 989 lines (prose says about 500 and 2,000)
  • number: mapping code 356, 370, 380 Go lines; 11, 31, 10 modified functions
  • number: translation tested on 8,750,468 Raft and 1,904,456 ZooKeeper traces
  • number: ten bugs, nine previously known, one new Redis Raft reconfiguration bug
  • limits: It handles fail-stop faults only. It assumes the supplied protocol is correct and the trace is faithful and ordered. It cannot prove correctness or catch livelock or deadlock. Humans wrote the translations and mappings. There is no public repository.
  • how much was read: most of it read
  • source; notes: Runtime Protocol Refinement Checking for Distributed Protocol Implemen; notes: Runtime Protocol Refinement Checking for Distributed Protocol Implemen; notes: Runtime Protocol Refinement Checking for Distributed Protocol Implemen
  • record id: ellsberg-2025
• FormalBench (2025)
  • title: Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference
  • what: Introduces FormalBench, a benchmark of manually validated Java/JML programs and semantic-preserving variants for evaluating LLM specification inference using deductive consistency, mutation-based completeness, robustness and self-repair metrics. FormalBench-Base contains 700 validated Java programs; 18 semantics-preserving transformations produce 6,219 diverse variants. OpenJML determines success/failure/unknown and Major generates mutants; completeness is the fraction of non-equivalent mutants rejected. The study evaluates eight open-source/proprietary LLMs under zero-shot, few-shot, chain-of-thought and least-to-most prompts, diagnoses syntax/quantifier/loop-bound errors, and tests targeted self-repair prompts.
  • number: 700 manually validated programs and 6,219 augmented variants
  • number: 14 LLMs reported in conclusion; core table covers eight models/settings
  • number: Best listed success: DeepSeek-V3 + LTM 16.6%; GPT-4o + LTM 15.0%; Claude-3.5 + LTM 15.4%
  • number: Robustness flip rates: DeepSeek-V3 27.2%, Claude-3.5 39.2%, GPT-4o 29.2%
  • limits: Mutation completeness is a proxy: equivalent mutants can evade detection (3 of 232 manually checked, 1.3%). Unknown verifier results are separated from failures but complicate metrics. The benchmark is Java/JML-specific; generated specifications may be syntactically invalid or semantically incomplete. Experiments omit OpenAI-o1 and DeepSeek-R1 due cost/access and use finite token/GPU budgets.
  • how much was read: most of it read
  • source
  • record id: formalbench-2025
• FormalSpecCpp: A Dataset of C++ Formal Specifications created using LLMs (2025)
  • what: # Whole-paper read: FormalSpecCpp: A Dataset of C++ Formal Specifications created using LLMs (authored by agents unless marked 🧑) Madhurima Chakraborty, Peter Pirkelbauer and Qing Yi; arXiv2502.15217v1,21February2025,5pages. The paper constructs105small C++ programs/contracts and translated tests using GPT-4-turbo, starting from verified Dafny-synthesis programs. Its evidence is compilation, finite tests and expert review of a subset. The introductory description of “verified C++ programs” is an author claim; no C++ deductive verification, verified translator or preservation theorem is supplied. [Q0,Q1,Q4,Q5,Q6,Q7] ## Construction, contracts and trust Only Dafny programs containing both preconditions and postconditions were selected. GPT-4-turbo translates their code and specifications; MBPP tests are separately translated to C++. Figure1 shows the Python→Dafny→C++ provenance and test/manual-inspection branches. This is translation from existing specifications, rather than an evaluation of inferring unknown C++ contracts. Figure2 asks the model to select signed/unsigned integer types, identify failure conditions, express preconditions with REQUIRE, loop invariants with assert, postconditions with ENSURE, and omit main. Both macros expand to assert. These are executable checks on reached states, not discharged proof obligations over all allowed inputs. They may also be disabled by the usual C++ assertion mechanism. [Q2,Q3,Q4,Q5,S2] The paper argues that translating verified Dafny specifications preserves correctness and avoids unnecessary constraints. That does not follow just from a verified source: integer domains, containers, evaluation order, exceptional behavior and the translation of the property itself also need a correspondence argument. The paper’s own manual review found semantically incorrect but syntactically valid specifications. The source Dafny verifier, selection, language mapping, generated tests and reviewer judgment therefore remain trust dependencies. There is no independent exhaustive audit of all105contracts in this paper or in this reading. [Q4,Q6,Q7,Q10] The paper contrasts unbounded Dafny integers with C++ fixed-width types, but incorrectly says C++ has “well-defined overflow behavior for signed integers.” Ordinary out-of-range signed arithmetic has undefined behavior. The saved C++ draft’s expression rules state that arithmetic-expression behavior is undefined when the mathematical result is outside representable values (with specified floating-point exceptions); unsigned modular arithmetic is distinct. This is a reader correction of a language claim, not an executed counterexample to every translated program. [Q10,L0] ## What the evaluation establishes The initial linked-test run had62programs pass all tests and43fail due to assertions, linking or compilation; 62/105≈59.05% is my arithmetic, not a reported semantic-correctness rate. Authors found misordered test arguments, incorrect expected outputs and unsigned-int/double mismatches. Compiling successfully did not exclude incorrect contracts or logical inconsistencies. One author reviewed/refined failing cases and a subset of successes; the paper gives no exact reviewed-subset size, final complete semantic-pass count, independent reviewer agreement or measured coverage. [Q6,Q7] TableI/§IV report105/105first-attempt program compilations,105files,$2.07program-translationcost(about$0.02/file),17.3average Dafny lines,33.1average C++ lines and91.2%growth. The text reports about27minutes for programs and an additional1.31/about15minutes for tests. These are historical generation/compilation figures, not105proved C++ implementations. There is no contradiction merely between first-attempt program compilation and later failing linked tests: they are different stages. The artifact makes the distinction concrete with g++ -std=c++11 -c for generated functions and linked executables for tests. No hardware, exact GPT snapshot/date, seed, repeated-run distribution, token totals or human-review time is given in the paper. [Q8,S2,S4,S7] The introduction proposes precision, recall, correctness, fine-tuning and inference-tool evaluation as uses. It reports none of those downstream comparative experiments. No baseline, prompt ablation, contamination study or evidence of scaling to large legacy C++ is supplied. The authors' efficiency/scalability language is broader than the measured105small-program conversion. Future work explicitly includes larger and more complex programs. [Q1,Q8,Q11,Q12] ## Pinned artifact and concrete cases The paper-linked GitHub repository was inspected at commit3f70080088d6d6ad3fe7923f94d2a91968cdc78c. Its README says the benchmark was published at MSR2025; I did not establish separate proceedings metadata. A nontruncated tree lists105with-contract C++ sources,105NoSpec sources and105test sources plus105same-basename executable entries. I did not download or execute those binaries. I read both READMEs, all four Python scripts, requirements, the human-readable conversion report, five program/test pairs (IDs3,14,101,105,106), and ID3's NoSpec counterpart. That is a targeted static sample, not an audit of the entire dataset. Upstream Dafny source and raw serialized experiment logs were not acquired/loaded. [S0–S18] The translation script calls model gpt-4-turbo with temperature0 and max_tokens2000, extracts a fenced cpp block, compiles an object, and can repair compilation errors for up to10attempts. The checked analysis report records105first-attempt successes,105total attempts,2.07 and no compilation errors, agreeing with TableI. The report is author-supplied evidence, not a reproduced run. The code’s cost tracker counts only the initial response, so hypothetical repair costs would be omitted; because this report says all attempts were first attempts, this observation does not establish an undercount for the reported run. [S2,S7] getTests.py uses temperature0/max_tokens500 and up to5attempts to extract a code block, then writes a test including the translated program. The script labels response.usage.total_tokens as input_tokens and also charges completion_tokens at output rates. Static accounting therefore charges completion tokens once at input rates as well as output rates; only the final successful response’s usage is returned, omitting failed extraction attempts. Actual request histories are unavailable, so I do not compute a corrected1.31total or assert that these two effects cancel. The runner counts process success/failure, but its error-category tests search str(CalledProcessError) without captured stderr, limiting reliable diagnosis of assertion/linker/compiler categories. Neither issue turns a successful test return into a proof. [S3,S4] The static cases show what the annotations mean. IsNonPrime(ID3) checks a divisor-search invariant and a quantified-by-loop postcondition; its tests cover2,10,35. KthElement(ID101) checks1≤k≤array size and returns arr[k−1]; three tests confirm specified positions, not sorted-order statistics. CountTrue(ID105) compares the running count with a recursive helper and excludes empty vectors. AppendArrayToSeq(ID106) preserves the input prefix and appended elements with explicit loops; it requires !a.empty(), commented “equivalent to a != null in Dafny.” Nonempty and nonnull are different conditions: an empty C++ vector is a valid value. Without the upstream Dafny source I flag the local comment/contract mismatch, rather than claiming a checked source-to-target counterexample. All five sampled tests contain three explicit input cases. [S8–S17] TriangularPrismVolume(ID14) accepts positive unsigned arguments, computes (base*height*length)/2, and asserts equality with that identical expression. The contract checks the same finite-width, integer-division expression, so by itself it does not establish unbounded arithmetic or fractional-volume semantics. Its tests use products with integral halves (10,8,6→240;3,2,2→6;1,2,1→1). IDs3/105/106 also execute additional loops or recursive helpers for assertions, which is actual checking work while assertions are enabled. No runtime-overhead measurement is reported. ID3's NoSpec file removes those checks; the removal script is textual pattern matching rather than a verified transformation. [S5,S8–S18] ## Contribution and read scope This adds an accessible LLM-assisted contract dataset and concrete translation/test failures to the specification-evaluation collection. My inference for evaluating proof/specification agents is to separate source verification, target compilation, finite testing, expert repair and target proof; inspect input-domain restrictions and arithmetic semantics; and distinguish repeated computations from independently adequate contracts. The paper has no distributed-protocol, Verus/Rust, TLA+ or proof-writing-agent experiment. [Q4,Q6,Q7,Q10,Q12] Read all5pages,15references,both figures including the full prompt image,TableI and all printed code; viewed all5page renders. Official arXiv and inventory PDFs are byte-identical. Read19pinned static text files with SHA256/Gitblobidentities, plus the cited C++ draft paragraph. No author code was executed/imported; no compiler, checker, prover or solver was run; no serialized scientific experiment results were loaded. Reported outcomes remain author-reported. - limits: 105smallDafny-derivedprograms;assertiontests/subsetexpertreviewnotdeductiveC++proof;signedoverflowclaimcorrected;targetedartifactsample - how much was read: most of it read - source - record id: `formalspeccpp-a-dataset-of-2025` - FVAPPS (2025) - title: Proving the Coding Interview: A Benchmark for Formally Verified Code Generation. - what: FVAPPS has Lean theorem statements that were auto-translated from APPS coding problems. They contain sorry placeholders (marks for missing proofs) that agents fill in. They come in three quality tiers. - number: 4715 problems; 2089 pass the unit-test-guarded tier (later paper reports 1083 guarded) - number: median 4 theorems per sample - number: manual audit of 100 samples labels too trivial, nontrivial, unprovable - number: 4,715 samples, 1,083 curated; Sonnet proves 30% - limits: Type-checking does not certify that a statement is adequate. The notes infer that the election contract in Fig. 2 is satisfied by a constant zero. The quality of the natural-language intent is not independently certified. - how much was read: most of it read - source; notes: FVAPPS - record id: `fvapps-2025` - HornStr (2025) - title: sfHornStr: Invariant Synthesis for Regular Model Checking as Constrained Horn Clauses. - what: HornStr synthesizes regular-language inductive invariants for string-theory CHCs using SAT-based DFA learning or L* active learning. Standard SMT-LIB2.6 string solvers check initiation,transition closure and bad-state exclusion; reachability enumeration supports active membership and inductive-counterexample handling. It connects parameterized-protocol regular model checking to existing string solvers and generates string-query benchmarks. Describes restricted unary-predicate problem,learning/refinement flow,DFA-to-regex translation,heuristic reachability cutoff,incremental solver sessions and separate membership/equivalence backend choice. Reports complete query-solver and protocol results,including active-learner timeouts,unlisted hard protocols and prose/table inconsistencies. No general completeness theorem or verified implementation is supplied. - number: 18 benchmark protocols (Table 2); SAT-based learner solved all 18, mostly in under a second; MU puzzle 11.01 s with a 3-state automaton - number: The active learner timed out on 6 of the 18 (Burns, Mux-Array, Eqdist, MU Puzzle, Water-Jug, Dining-Crypt) - number: Z3-noodler solved all 848 non-incremental equivalence queries; cvc5, Z3, Z3-alpha solved 403 to 457 - number: Generated more than 30000 string constraints (abstract) or more than 10,000 (Sec 5); the paper gives both numbers - limits: §2: one unary unknown predicate over strings; sought solutions regular languages. General CHC definition allows more predicates/nonlinearity, but that syntax is not evidence that implemented solver handles all general string CHCs. Symbolic alphabets and broader CHC support deferred in §5. §§2.1–2.2: safety expressed by initial inclusion,transition closure,error exclusion; configurations are modeled words. Correctness of protocol→CHC modeling/automatic translation assumed,not mechanized here. Arbitrary number of processes does not establish real Rust implementation correctness. §2.1: reachable/bad-predecessor languages can be nonregular while a separating inductive invariant is regular. Failure to find a regular invariant need not mean unsafe; no complete procedure for all infinite-state systems. §3: SAT unsat means current DFA size cannot fit examples and triggers larger size; string-query unsat means no violation of the particular checked clause. These are different meanings,not contradictory CHC conclusions. §3: active membership enumerates reachable words and may answer negative after same-length exhaustion or fixed cutoff. Explicit heuristic inspired by length preservation; bounded failure to find a word does not generally prove unreachable in length-changing rewriting. Could impair learning/completeness. Final all-clause validation remains distinct,conditional on solver correctness and handling of unknown/timeouts. §3: DFA→regex uses Brzozowski/McCluskey state elimination; induction counterexamples can be converted to positive/negative via reachability following strict-but-generous teacher. No in-paper full correctness/termination proof,complexity bound,formalized checker or proof-producing string backend. §3/Fig1: learners exchange regex equivalence queries and reachability membership queries through independent interfaces. Each CHC can have dedicated incremental solver thread with push/pop; optional single solver. Trust includes parsers,translation,learners,regex conversion and SMT solvers; artifact not executed. §4.1: query families have unequal counts: incremental523membership/396equivalence versus nonincremental30902/848. Raw aggregate runtimes cannot isolate causal per-query incrementality benefit without matched workloads. Table excludes timeouts from times; lower solved-time can simply reflect fewer solved queries. §4.1: prose calls523membership queries an average109.7s and says other solvers slightly higher runtimes. Table1 labels Time(s),caption solved time;Z3 33.7,cvc5 97.8,Z3-alpha86.4 are lower than109.7. Preserve table and inconsistent prose;109.7s as per-query average would exceed stated30s timeout. No silent correction or invented timing definition. §4.1: visual PDF p.8 confirms disequality x≠c; text extraction drops stroke and yields x=c. Regular-expression simplifier/flags marginal or harmful overall,not robust improvement;no quantitative ablation table. §4.2:18listed protocols all solved by SAT;active12solved/6timeouts. Kanban/German timeout60s in prose but absent from Table2;therefore SAT success on every listed row does not mean all benchmark protocols solved. Full-suite count and which learners timeout on unlisted protocols not supplied. §4/Table2: automaton sizes and learning seconds;no printed invariant automata/independent certificate or unsafe-trace count per row. MU3states/11.01s is reported discovery,not this reader rerunning solver or inspecting artifact invariant. First-solver claim is authors’ claim,not independently established priority. §4:IntelCorei7-10510U1.8GHz/16GB/Windows11.30s query and60s protocol limits;solver revisions,repeat counts/variance,full timeout overhead,independent baseline RMC comparison and frontend correctness unreported here. §§abstract/5:abstract says more than30000newQF_S constraints;conclusion says more than10000. Compatible lower bounds but inconsistent specificity;do not sum query categories as disjoint unique files. Liveness,bisimulation,further CHCs future work. No Verus/Rust/TLA+ or LLM experiments. Full version[30]arXiv2505.15959 and artifact[31]Zenodo15195124 contain deferred detail;not read/executed. This is proceedings-paper read only. - how much was read: most of it read - source; notes: ## HornStr (2025) - record id: `hornstr-2025` - LAW (2025) - title: The LAW theorem: Local Reads and Linearizable Asynchronous Replication - what: The paper proves that no replicated register can have all three of these: a local read (a read with no messages), linearizability, and a guarantee that operations finish despite async delays and one crash. It designs almost-local reads (batched read validation) in Hermes, ZAB and Raft variants. - number: Raft-ALR 2.6x baseline throughput at 2% writes, 1.8x at 5%; up to 156 million requests/s - number: ZAB-ALR within 4% of ZAB at 1% writes; Hermes-ALR within 11% at 1% writes - number: 5 CloudLab servers, one million keys; reads batched up to 128 - limits: A human wrote a mathematical proof about a mathematical register (deterministic servers, one active operation each, at most one crash). The correctness of the read algorithms is argued in prose. The Raft baseline relaxes linearizability, so the performance comparison is an upper bound. There is no LLM, no code proof and no person-hours. - how much was read: most of it read - source; notes: Added related work: The LAW theorem: Local Reads and Linearizable Asyn - record id: `law-2025` - Lilo (2025) - title: Lilo: A Higher-Order, Relational Concurrent Separation Logic for Liveness - what: Lilo is a Coq logic for the liveness of shared-memory programs. It uses obligations (promises to finish) and progress credits. The authors give proofs for a spinlock, a ticket lock, Treiber and elimination stacks, and infinite message passing. - limits: Stratified propositions limit contracts that refer to themselves. Logically atomic triples for the elimination stack are future work. It is not a result about networks or crashes. - how much was read: most of it read - source; notes: Iris: Trillium, Lilo, Lawyer - record id: `lilo-2025` - LLM document-to-temporal-spec extraction (2025) - title: Extracting Formal Specifications from Documents Using LLMs for Automated Testing - what: Builds a dataset of 603 temporal specifications from 37 ArduPilot, PX4 and Autoware documents and evaluates GPT-4o, Claude-3.5-Sonnet and Llama-3.1-405B extracting formal requirements. A two-stage annotation-then-conversion pipeline improves coverage and accuracy over direct end-to-end prompting. Experts cross-validated ground truth; prompts first mark requirement-bearing sentences and then convert annotations into temporal logic, reducing oversimplification and hallucinated clauses. The paper quantifies false positives and studies a DeepSTL non-LLM conversion baseline. Documents average 2,966 words; each model runs three times and unique outputs are aggregated. - number: 603 validated specifications from 37 documents - number: End-to-end accuracy: Claude 51.7%, GPT-4o 47.1%, Llama 45.4% - number: Annotation-then-conversion best accuracy 71.6%; +14.0 percentage points average and +29.2% correctly extracted - number: Best false-positive rate 14.5%; each LLM run three times - limits: Ground truth may be incomplete or contain expert errors; temporal formulas require interpretation of informal documents. LLMs oversimplify dense requirements and fabricate plausible but unsupported conditions. Dataset is limited to three open-source autonomous-system projects and temporal logic; extraction quality does not prove implementation correctness. - how much was read: most of it read - - record id: `llm-document-to-temporal-spec-extraction-2025` - LMGPA (2025) - title: Towards Language Model Guided TLA+ Proof Automation - what: An LLM splits hard TLAPS proof obligations into smaller claims. Symbolic provers check them. The invariant is given as input. - number: 26 inductiveness proofs of distributed protocols plus 93 math theorems (119 total) - number: Distributed protocols (Table 1 p. 13): symbolic-only 38.5%; DeepSeek-V3.2-Exp 50.0% with decomposition (13/26), 0% direct; Claude 3.7 Sonnet 42.3% vs 15.4% direct; GPT-5 42.3% vs 3.8% direct - number: Runtime DeepSeek 9h 14m vs Claude 1h 32m - number: Four decomposition attempts per claim; TLAPS timeouts Z3 5 s, Zenon 10 s, Isabelle 30 s - limits: It tests proving a supplied invariant. It does not test finding one or reading vague requirements. Training leakage is possible from public protocol proofs (Section 4.2). The suite leaves out both learning switches, chord and hybrid broadcast. - how much was read: most of it read - source; notes: LMGPA: prove supplied invariants rather than discover them; notes: TLAPS proof generation grew beyond the January 2025 toy study - record id: `lmgpa-2025` - Maeslant BesW (2025) - title: A Complete Formal Specification and Verification of the BESW Software Control System of the Maeslant Storm Surge Barrier - what: Translates the 15,000-line PLC control design of the Rotterdam storm surge barrier's pump, valve and door controller into an mCRL2 model, then model-checks 40 requirements written as modal mu-calculus formulas. - number: mCRL2 model about 5,000 lines, 4.98e14 states - number: 40 properties: 36 valid at once, 3 valid after rewording, 1 invalid (deprecated requirement) - number: Each property verified within 40 minutes on 2x24-core, 1.5 TB machine - number: Found one design issue: with pump faults, more pumps can run than the power budget allows - limits: Water levels are discretised, not all pump and valve fault combinations are checked, status and log messages and the BOS data protocol are left out. The real model is confidential, so it cannot be reused or re-checked. - how much was read: most of it read - source; notes: ## Maeslant storm surge barrier control software in mCRL2 (2025) - record id: `maeslant-besw-2025` - miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward (2025) - what: Re-checks the miniF2F Lean statements against the original competition questions and corrects them. Separates simplified tasks from tasks faithful to the competition. - number: 488 problems; over 300 Lean 4 statements corrected; 16 problems with no solution - number: Herald test accuracy automated judge 97.5% vs human 62.7% on v1 - number: Kimina+Goedel-V2-7B test 49.2 / 69.7 / 61.5 for v1/v2s/v2c; validation 54.9 / 70.1 / 62.3 - number: Pipeline about 36% (40% on the original) vs 97% and 69% for the two stages alone - limits: About mathematics, not software. Judging fidelity needs a human. - how much was read: most of it read - source; notes: miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forw - record id: `minif2f-lean-revisited-reviewing-2025` - MongoDB conformance blog (2025) - title: Conformance Checking at MongoDB: Testing That Our Code Matches Our TLA+ Specs - what: This 2025 retrospective describes MongoDB engineers testing whether C++ implementations conform to TLA+ specifications. It contrasts model-generated tests with trace checking, reports practical obstacles such as mapping implementation state back to the model, and describes a Raft-derived replica-set model with safety and liveness properties. - number: The post revisits two MongoDB products and work first tried in 2020. - number: A month to figure out how to instrument MongoDB - number: 4913 tests reached 100% branch coverage - limits: Trace checking and generated tests are incomplete samples; the post says completeness is usually impossible. - how much was read: most of it read - source - record id: `mongodb-conformance-blog-2025` - Neural model checking with invariants (2025) - title: Let a Neural Network Be Your Invariant - what: For hardware designs, one small neural network is trained to be both an inductive invariant (a set of states that contains all reachable states and is closed under steps; this proves safety) and a ranking function (a number that drops each time something bad could recur; this proves liveness). A solver then checks the network exactly, so a yes answer is a real proof. Training uses a constraint solver (MILP) rather than gradient descent. - number: 634-task benchmark: 11 SystemVerilog designs; 206 safety, 219 liveness, 209 safety+liveness tasks - number: solved: 190/206 safety (92%), 192/219 liveness (88%), 172/209 safety+liveness (82%) - number: nuXmv solved 82% of liveness and 74% of safety+liveness tasks; rIC3 50% and 61%; ABC 36% and 46%; prior neural method NMC'24 79% of liveness - number: pure liveness: faster than the best other tool on 66% of tasks, 10x faster on 46% - limits: Only finite-state hardware-style systems (no arrays or strings of parametric length, no procedure calls or dynamic data); the benchmark is the authors' own, built from textbook-level designs, so results may not carry to other workloads; relies on sign-activation feed-forward networks. - how much was read: most of it read - source; notes: Let a Neural Network Be Your Invariant (2025) - record id: `neural-model-checking-with-invariants-2025` - ParCleanse (2025) - title: Validating Network Protocol Parsers with Traceable RFC Document Interpretation - what: GPT-4o extracts packet layouts and constraints from RFCs, with links back to the source text. Z3 generates valid packets and packets that break exactly one rule, and real parsers in C, Go and Python run them. An LLM diagnoses whether each failure is a spec error or an implementation error. - number: Table 2 precision/recall: Total 100/100 99/95 94/91 98/82 73/44 - number: Table 3: 69 unique, 68 new, 36 confirmed bugs - number: golden formats: Babel 171 lines/5 h; BGP-4 164 lines/6 h - limits: It recalls dependent rules poorly. Every discrepancy was checked by hand. It covers input-validation safety, with no liveness. - how much was read: most of it read - source; notes: Validating Network Protocol Parsers with Traceable RFC Document Interp - record id: `parcleanse-2025` - PSMBench (2025) - title: PSMBench: A Benchmark and Dataset for Evaluating LLMs Extraction of Protocol State Machines from RFC Specifications - what: A benchmark of 14 protocols with human-annotated state machines. It scores nine LLMs on extracting state machines from RFC text. - number: 14 protocols, 1,580 RFC pages, 108 states, 297 transitions - number: annotation about three days per protocol - number: best state F1 0.715 (DeepSeek-V3), transition F1 0.378 - number: 14 protocols, 1,580 RFC pages, 108 states, 297 transitions; annotation about three days per protocol - limits: It scores agreement with the annotations, not protocol safety. The annotations themselves need auditing, since RFC2TLA+ finds missing behaviors. Settings in the appendix give other scores. - how much was read: most of it read - source; notes: PSMBench / RFC2PSM: extraction accuracy, with human ground truth; notes: PSMBench / RFC2PSM: extraction accuracy, with human ground truth - record id: `psmbench-2025` - QSM-Cutoff (2025) - title: QSM-Cutoff: Systematic Derivation of Quantified Cutoff Formulas for Distributed Protocols - what: Enumerates reachable finite states using symmetry. Minimizes their logical description. Grows the domain sizes until a saturation test passes. - number: 19-model evaluation (Section 7, pp. 278-280), e.g. toy consensus forall: node=5, quorum=10, value=3 - number: Pages 263-286 - number: 19 protocols - number: 6000 s/run - limits: That the formula is a true reachability invariant for every size is still a conjecture. Ordered protocols such as Paxos and Bakery are future work. - how much was read: most of it read - source; notes: QSM-Cutoff: a full 19-model 2025 set - record id: `qsm-cutoff-2025` - QuickSilver / D-Escalate / Venus (2025) - title: Automating Formal Verification of Distributed Systems via Property-Driven Reductions - what: Sound reductions from parameterized models (arbitrarily many processes, large data domains) of agreement-based protocols to small finite models. QuickSilver, D-Escalate and Venus check them. The developer writes a Mercury/Coalition model and a safety formula. - number: Table 3.1: 12 Mercury models, 32-99 lines, cutoffs 2-5, 0.059-736.417 s - number: Table 4.1: 23 Coalition cases, 78-126 lines, one cutoff 6 - number: Table 5.1: 10 Venus cases, 34-79 lines, 0.202-1.620 s - number: means over 10 runs - limits: It covers safety of the parameterized model only. The termination of the agreement primitives is assumed. One cutoff was computed manually. Human modeling time is not quantified. - how much was read: most of it read - source; notes: Automating Formal Verification of Distributed Systems via Property-Dri - record id: `quicksilver-d-escalate-venus-2025` - Remix (2025) - title: Multi-Grained Specifications for Distributed System Model Checking and Verification (Remix) - what: Refines coarse TLA+ protocol actions into detailed ones where concurrency matters. It mixes levels of detail to keep model checking practical, and checks the model against implementation executions (ZooKeeper). - number: six severe bugs violating five types of invariants - number: Six deep ZooKeeper bugs found and fixes verified - number: abstract: found six severe bugs that violate five types of invariants - number: six severe bugs - limits: The notes infer that matching the implementation and fixing bugs found by the model checker do not validate the chosen client guarantee, and do not prove liveness for unbounded cases. - how much was read: most of it read - source; notes: Choosing top-level distributed guarantees and trusted boundaries; notes: Appendix B: Concrete systems and theorem scope - record id: `remix-2025` - Reusable Formal Verification of DAG-based Consensus Protocols (2025) - what: # Reusable Formal Verification of DAG-based Consensus Protocols (authored by agents unless marked 🧑) Nathalie Bertrand, Pranav Ghorpade, Sasha Rubin, Bernhard Scholz and Pavle Subotić, arXiv2407.02167v2,18March2025. The inventory year2024 identifies the original submission; this read uses the retained2025revision. The authors report TLA+/TLAPS safety proofs for five protocol models by combining three DAG-construction components with two ordering components. The five are DAG-Rider, Cordial Miners, Hashgraph, Eventually Synchronous Bullshark and a minor variant of Aleph. The result concerns Integrity and Total Order; eventual delivery, probabilistic progress and implementation correctness remain outside it (Q0,Q1,Q2,Q12). ## Reading scope and source identity Read all21PDFpages, sections1–6 and51reference entries, all three figures including both panels of Figure2, seven listings and numerical Table1. Individually viewed pages6–16, covering the figures, listings, refinement mapping and numerical table. The paper refers to “Appendix ??” but includes no appendix. References were read as bibliography entries, not recursively read papers. Acquired the paper-linked repository's complete nontruncated tree and22static blobs at9e4c9629eb283c7c34b6ffcf7956bc72a3e26e5d,22December2024, the latest retained commit before this paper revision. This selects a historical version without establishing that it produced the reported timings. Whole reads cover README and the five component Specification.tla files. Proof files and complete-protocol sources are inspected only in explicitly recorded ranges. Acquisition of a file does not imply its whole read. No author code, specification, checker or solver was executed/imported; no serialized experimental data or model was loaded. ## What is proved and how composition works A vertex embeds its entire rooted causal history, rather than only an opaque identifier. Thus two processes possessing the same vertex have identical histories for it. A separate anchor-agreement property requires processes that have selected anchors for a frame to select the same anchors. Delivering each frame's previously undelivered causal-history vertices in a fixed deterministic order then prevents duplicate delivery and gives consistent delivery order. A lagging process may deliver a shorter prefix; the proof is not simultaneous equality of local delivery states (Q2,Q3). The decomposition is three construction models, reliable structured, unreliable structured and unreliable unstructured, plus Global Perfect Coin (GPC) and Virtual Voting (VV) ordering. Reliable construction ensures one agreed vertex per position. Unreliable structured construction allows equivocation and uses ratification to agree on a vertex at a position. GPC ordering separately agrees on leader positions; combining these properties establishes anchor agreement. Hashgraph first converts its unstructured events to a structured witness DAG. Reference consistency constrains witnesses' predecessor references; VV establishes agreement on famous witnesses (Q4,Q6,Q7,Q8). The full protocol maps constants and new state variables into an INSTANCE of the ordering specification. Each construction action performs an appropriate ordering action or leaves its mapped variables unchanged. Protocol-specific refinement obligations remain necessary: five reusable components do not eliminate the proofs connecting five concrete models to them. DAG-Rider and ES Bullshark share reusable components but differ in wave lengths and commit/leader rules. Aleph is explicitly a variant; neither the paper nor its one-line repository README gives a precise delta against every published Aleph rule (Q4,Q9,C6). ## Assumptions and abstractions The system explanation assumes signed identities and fewer than one-third Byzantine processes. Reliable broadcast is weakened to integrity because validity and eventual agreement support liveness. Its model retains the first broadcast for each creator/round; the authors argue a faulty process's chosen delivered message could equivalently have been sent first. This is an abstraction argument to retain in correspondence work, not a separately measured implementation equivalence (Q1,Q5). Unreliable communication copies a vertex from another local DAG; Byzantine participants can insert arbitrary modeled vertices. Procedures retain input/output behavior while omitting internal loop states. GPC becomes an arbitrary fixed function shared by all processes: randomness, unpredictability and fairness are absent. Agreement of the function's answers suffices for the stated safety argument; it supplies no probability-one termination theorem or implementation of a cryptographic coin (Q5,Q7,Q8). The retained component models use finite symbolic bounds on waves/rounds or event IDs, with no fixed numerical test instance. Proof claims quantify over their parameters; a single TLC run at one bound would not establish that claim. The record does not independently establish an infinite-execution refinement beyond these bounded model families. The VV component incorporates its reference-consistency premise directly into both the prestate and poststate of Next. Its source proof of that premise therefore depends on the action restriction; a concrete protocol still must justify the mapping (C1–C5). ## Consequential findings in the retained artifact The printed self-referential VertexSet listing is explanatory shorthand. ReliableStructuredDAGconstructionSpecification.tla instead constructs vertices recursively by decreasing round with UntilRoundVertex. That resolves the apparent circular-definition concern for this component. However, its received/added-vertex guards access v.edges, while the actual constructed records contain strongedges and weakedges. This is a static field mismatch in the saved model; no checker error or reachable protocol failure was observed (C1). In the unreliable structured component, Approves(v,u) requires Observes(v,u) and forbids observing every vertex with u's source and round. Taking that quantified vertex to be u yields both Observes(v,u) and its negation. Thus the literal definition admits no approval. With n-f positive under3f<n, ratification requires a nonempty set of approving sources and cannot occur under this reading. Since ratified vertices initialize to Nil, the displayed agreement property can hold without any ratification progress. This is a reader derivation from the retained definitions, not a claim that the real Cordial Miners protocol fails (C2). The unstructured component's DonotSeeFork(v,u) has the same self-including universal condition. Taking x=u contradicts its Path(v,u) conjunct. Its Sees predicate consequently cannot witness a seen ancestor under the intended reflexive Path interpretation. This is another source-definition concern to resolve before trusting correspondence to actual Hashgraph behavior; the paper excludes liveness and no safety counterexample is established (C3). VVOrderingProofs.tla contains several named lemma statements without proof text. Most concretely, intInv14 asserts, for integers a,m, that a>=m and m+a>2f imply m>f. The permitted assignment f=1,a=2,m=1 makes both premises true and its conclusion false. The file cites this lemma in its Inv1 preservation proof. The nearby intadd statement has the plausible intended conclusion a>f with strict a>b. This suggests a proof-support defect or mistaken statement in the retained artifact; it does not show a checker accepted a false theorem. The actual preservation goal is a majority-count fact that can be mathematically true even though this cited general lemma is false (C7,C8). The same proof preamble asserts, without accompanying proof text, that every witness has more than2f predecessor sources in the preceding frame. Combined literally with its positive finite frame range and predecessor-frame condition, that leaves no nonempty witness set: a witness of minimum frame would require an earlier witness. This preamble should not silently become an established domain theorem. A simpler nonvacuity concern appears in the VV specification itself: all witness-DAG cells initially are empty, while adding a witness requires more than2f predecessor sources already present. With f nonnegative and no genesis exception, no first witness can be added under the intended typed interpretation. Whole proof closure and tool handling of unproved declarations have not been replayed. These artifact findings limit independent confirmation of the paper's mechanized-proof claim; they do not refute the high-level decomposition or establish a reachable fork (C5,C7). ## Evaluation and significance Table1 reports, in order reliable structured/unreliable structured/unreliable unstructured/GPC/VV: specification lines403/160/230/272/136; invariants6/6/7/10/18; proof lines460/594/554/822/2120; maximum proof depths10/9/8/9/13; maximum degrees7/8/7/7/11; leaf obligations633/895/665/1302/3316; and checking seconds49/68/74/125/651. Environment:2.10GHzCPU,8GBRAM,Windows11,TLAPS1.4.5. These are author-reported component checks, not reader replications or total timings for every full-protocol refinement. No repeated-run variability or controlled comparison appears (Q10). The authors report approximately14person-months across five people. Their “almost half” proof-effort claim corresponds to reducing an illustrative ten separate construction/ordering components to five reusable ones. There is no controlled person-time baseline or ablation demonstrating a50% labor saving; refinement work and component complexity also differ. A reusable component count is concrete, while a measured causal labor reduction remains unestablished (Q0,Q4,Q10,Q12). This is directly relevant to reusable TLA+ specifications and distributed-protocol proofs. It has no Verus/Rust artifact or LLM-agent experiment. A reader proposal for evaluating proof agents is to score specification correspondence, nonvacuous progress examples, closure of helper lemmas and refinement obligations separately from reported leaf-obligation throughput. The retained field/predicate/lemma issues make those distinctions concrete without treating static concerns as observed solver results. - number: approximately 14 person-months across 5 people - number: Table 1 spec lines 403/160/230/272/136; proof lines 460/594/554/822/2120; obligations 633/895/665/1302/3316; TLAPS time 49/68/74/125/651 s - number: TLAPS 1.4.5 on 2.10 GHz, 8 GB - number: Reuse reduces proof effort by almost half - limits: Safetyonly;abstractbroadcastandcoin;finiteparameterizedhorizons;implementationrefinementfuturework;retainedsourcefield/predicate/helperlemmafindingslimitindependentconfirmation;nocheckerreplay - how much was read: most of it read - source; notes: Related addition: Reusable Formal Verification of DAG-based Consensus - record id: `reusable-formal-verification-of-2025` - Reusable TLA+ Communication Primitives for Modeling and Verifying Distributed Systems (2025) - what: A PlusCal Echo protocol built over reusable channel abstractions (perfect, fair-loss, stubborn). TLC checks eventual delivery, FIN termination and that no message is invented. - number: MaxCopies set to two; only Property 3 held under duplication - number: no state counts, timings, LOC or hours - limits: The checks are bounded model checks. The user handles nondeterministic drops and eventual delivery. The perfect link is FIFO, which is stronger than usual. - how much was read: most of it read - source; notes: Reusable TLA+ Communication Primitives for Modeling and Verifying Dist - record id: `reusable-tla-communication-primitives-2025` - Seed-Prover 1.5 (2025) - title: Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience - what: A model trained with reinforcement learning (trial and reward from the Lean checker) to prove theorems in Lean by calling Lean, a library search and Python as tools, plus a workflow in which a second model sketches a proof as small lemmas that the prover then fills in. Measured on math benchmarks. - number: PutnamBench 580 of 660 (87.9%), Fate-H 80%, Fate-X 33%, CombiBench 48% - number: Putnam 2025: 11 of 12 solved within 9 hours (budget up to 40 H20-days per problem); IMO 2025: 5 of 6 - number: Agent-only run (pass@8x8): Putnam 359 of 660 vs 331 for the earlier 18-H20-days-per-problem workflow - number: RL training accuracy rose from about 50% to nearly 90% over 1200 steps; average tool calls fell from about 15 to 10 - limits: Math only (competitions, algebra textbooks), not programs or distributed systems. The authors say CombiBench has significant formalization errors, and several Erdős results were simplified or trivial versions caused by mis-formalized statements; the full training recipe and data are in-house. - how much was read: most of it read - source; notes: ## Seed-Prover 1.5 (2025) - record id: `seed-prover-1-5-2025` - SpecVerify (2025) - title: Supporting Software Formal Verification with Large Language Models: An Experimental Study - what: SpecVerify converts natural-language requirements and implementation code into an intermediate formalized requirement/test design, then generates C assertions checked with ESBMC. It compares Claude-based verification with FRET/CoCoSim and SLDV on Simulink-generated aerospace C code. Analyzes variable mapping and precondition/function/postcondition correspondence, gives triple-redundant voting and missing aircraft-weight-assumption cases, and traces a floating-point median-selection bug that rational arithmetic missed. Counterexamples are replayed against code when results differ. The authors report a mistaken +1-to-0 connection in a prior REG model. - number: 9 systems; 58 requirements; 41–251 C lines - number: 46/58 logical equivalence (79.31%) - number: Claude+ESBMC completion 27/58 (46.5%) - number: Table V false negatives: CoCoSim 6; Claude+ESBMC 2 - limits: Nine retained systems contain only 41–251 C lines; one original benchmark cannot be converted. Benchmark code is generated and structured; larger manually written programs are untested. The logical-equivalence assessment is manual and can oversimplify temporal relationships. LLM and verifier effects are confounded. ChatGPT model naming varies across the source; Llama 3.1 8B failed tasks and was excluded. Table II/full-automation claims coexist with a stated need for considerable manual checking of generated code. Arithmetic inconsistencies remain in the source: 29/58 undetermined is labeled 39%; Table V false negatives are 6 versus 2, while the conclusion claims 6 fewer. These are source issues, not corrected measurements. - how much was read: most of it read - source; notes: Supporting software formal verification with LLMs (2025) - record id: `specverify-2025` - Taming Floating-Point Rounding Errors with Proofs (2025) - what: Overview of formal verification for NASA avionics floating-point software, distinguishing an incorrect real-number requirement from rounding errors in a finite-precision implementation. Reports CPR verification with Frama-C/PVS/Gappa, PRECiSA certified error estimation using abstract interpretation and Kodiak, and ReFlow generation of annotated C with warnings for potentially unstable branches. Summarizes earlier results; supplies no new detailed proof or benchmark. The entire publication is an extended abstract. Records its actual toolchain stages, trace-wise symbolic versus numerical certificates, warning semantics, expert manual versus automated verification scope, concrete avionics cases, cited-source pointers and explicit future-work status of LLM-generated-code verification. Does not upgrade an overview into an independent check of cited systems. - limits: p1: Original CPR operational requirements reportedly failed intended precision even over exact reals. Tightened real-number requirements and a proved double-precision implementation are separate results. The overview supplies neither numeric preconditions/tolerances nor a counterexample position; source[6] VSTTE2017 and source[19] FM2018 hold the detailed analyses. No guarantee for the unmodified original requirements follows. p1: Improved CPR expressions reduce numerical complexity; double-precision correctness is attributed to Frama-C/PVS/Gappa. The early workflow requires expertise and is not fully automatic. Arithmetic format alone does not establish correctness; relevant input ranges and semantics are not reproduced here. p1: PRECiSA abstracts collecting semantics, builds symbolic error expressions along traces and invokes Kodiak for sound numerical enclosures. Externally checkable certificates cover both symbolic and numerical error estimates, relying on PVS floating-point and branch-and-bound formalizations. Certificate format, proof size, checker version, trusted components, coverage of exceptional values and termination are not specified in this overview. p2: ReFlow extracts floating-point C from a PVS real-number specification, emits warnings where control flow may differ due to rounding and generates ACSL annotations linking the two. A warning is a disclosed possible divergence; this statement does not establish unconditional branch agreement or exact real-number equality for every input. No detailed unstable-branch example, emitted C/ACSL or warning-return contract appears. p2: Frama-C generates PVS verification conditions; ad hoc PVS strategies discharge them automatically in the described toolchain. The user-expertise claim concerns these supported generated programs. The overview reports no corpus, supported syntax/loop bounds, failure rate or theorem proving timings that would justify universal automatic verification of arbitrary C. p2: Applications are winding-number point-in-polygon for geofencing and a fragment of DAIDALUS. This is not verification of all DAIDALUS, all ADS-B communication, complete aircraft software, a distributed fault model or system liveness. p2: LLM-generated code and quantized neural networks are future challenges, not an implemented/evaluated proof-agent or verified neural inference result. No Verus, verified Rust or TLA+ system is presented. Three-page invited-talk overview has no original formal statements/proof scripts, measurements, quantitative error table, performance comparison, source audit or reproduced artifact check. Tool guarantees are author reports of cited results. - how much was read: most of it read - source - record id: `taming-floating-point-rounding-2025` - TLA+ Examples issue 174 (2025) - title: Spec claims to check (safety-prop) UnforgLtl but config actually also checks (liveness-props) CorrLtl and RelayLtl - what: An open issue in the TLA+ Examples repository flags a mismatch between a Byzantine broadcast specification's stated safety property and the configuration's additional liveness properties. It points to the exact TLA+ and configuration lines. - limits: The issue is a repository-maintenance report; it does not say whether the extra properties pass or fail, only that the configuration checks more than the description claims. - how much was read: most of it read - source - record id: `tla-examples-issue-174-2025` - TLAiBench (2025) - title: tlaplus/TLAiBench - what: 9 puzzle descriptions (for example DieHard, DiningPhilosophers, TowerOfHanoi) and 6 gold specs. A script asks an LLM to translate each description, then runs the syntax check and TLC. - number: 9 puzzle descriptions - number: 6 gold specs - number: inspected commit of 6 Feb 2026: ten puzzle descriptions, six gold models - number: no distributed implementation tasks; dining philosophers has no gold model - limits: No results are reported. Checking against the gold spec by refinement is only stated as a grading idea, and is not run at scale. - how much was read: most of it read - source; notes: Additional tools, not counted as papers; notes: TLAiBench - record id: `tlaibench-2025` - Tunable Automation in Automated Program Verification (2025) - what: Adds a `broadcast` mechanism to Verus so users choose which universally quantified facts the SMT solver may use, per module, function or proof block. It measures how extra facts about Seq/Map/Set/Multiset change proof hints needed and verification time on four Verus projects. - number: 49 Dafny-prelude lemmas added as ambient facts (Seq 25, Map 2, Set 10, MultiSet 12) - number: asserts after minimization: IronKV 268 to 245, Splinter 1158 to 1130, Anvil 343 to 336, CapybaraKV 449 to 415 (2% to 9% fewer) - number: more than half the original asserts were not needed - number: 98.2% of IronKV functions verify within 2x of original time; worst slowdowns 3x to 19x - limits: Assert count is called a very rough proxy for automation; the minimizer is a linear scan, not optimal, and ignores lemma calls. all_triggers did not scale to the other projects (many verification failures). - how much was read: most of it read - source; notes: Tunable Automation (2025) - record id: `tunable-automation-in-automated-2025` - Veil (2025) - title: Veil: A Framework for Automated and Interactive Verification of Transition Systems. - what: It embeds distributed transition systems in Lean. It sends first-order proof obligations to SMT solvers and leaves the rest as interactive Lean goals. - number: 16 evaluated cases incl. Rabia, SCP, Suzuki-Kasami, Vertical Paxos - number: 1704 non-empty, non-comment lines of Lean; 85 actions and 185 invariants; about 100 lines per file - number: 16 benchmarks, 1,704 Lean lines, 85 actions, 185 invariants - number: Veil verifies 16/16; 14/16 under 15 s and 12/16 under 10 s - limits: It evaluates checking supplied invariants, not finding them. It is not an LLM agent. All its guarantees are safety. The notes record no agent success rate. - how much was read: most of it read - source; notes: Veil: proofs in Lean with automatic first-order checks; notes: Veil: a promising proof interface, with no agent rate established here - record id: `veil-2025` - Vericoding benchmark (2025) - title: A Benchmark for Vericoding: Formally Verified Program Synthesis - what: Fixed formal specs in Dafny, Verus and Lean. Agents fill in implementation and proof holes. Validation blocks bypasses (assume, false/sorry), and fixed templates prevent changes to the spec. - number: 12,504 specs: 3,029 Dafny, 2,334 Verus, 7,141 Lean; 6,174 new/translations - number: about 9% of sampled successful specs too weak and another 15% had poor translations - number: 12,504 specs (3,029 Dafny, 2,334 Verus, 7,141 Lean) - number: Success: 82% Dafny, 44% Verus, 27% Lean - limits: Quality statistics come from five sampled successes per language and source, not from all items. The cleanliness score is a heuristic. A checker establishes the contract that was submitted, even if it is lossy or weak. - how much was read: most of it read - source; notes: A Benchmark for Vericoding - record id: `vericoding-benchmark-2025` - VerifyThisBench (2025) - title: VerifyThisBench: Generating Code, Specifications, and Proofs All at Once - what: Models generate code, spec and proof together for VerifyThis competition challenges. They use seven tools: Dafny, Why3, Frama-C, Verus, VeriFast, VerCors and TLA+. Each model gets five attempts with checker feedback. The XS variant removes parts of the verified submissions. - number: 41 challenges / 154 tasks; XS total 580 (226 code, 233 spec, 121 loop); 481 is stale - number: original human contest: 90 minutes per challenge - number: 41 challenges, 154 tasks, 580 sub-tasks - number: o3-mini under 4% zero-shot, 9.37% after five feedback rounds - limits: Alignment confidence is the model's self-assessment (the paper calls it an 'optimistic upper bound'). There is no independent exhaustive oracle for alignment. The notes did not establish that assume, admit or axiom is enforced. TLA+ gets no scoring for fairness or liveness. The human contest judges correctness, completeness, elegance and automation. - how much was read: most of it read - source; notes: VerifyThisBench / XS; notes: VerifyThis: human assessment supplies an intent layer - record id: `verifythisbench-2025` - VeruSAGE (2025) - title: VeruSAGE: A Study of Agent-Based Verification for Rust Systems - what: 849 proof-writing tasks from eight verified systems (IronKV, Anvil, Atmosphere, Vest and others). A proof body is removed, and an agent loop with a 20-minute budget must restore it. - number: Table 5 model totals: o4-mini 41%, GPT-5 55%, Sonnet 4 64%, Sonnet 4.5 81%; Sonnet 4.5 with plain Copilot proves 688/849 (80.9%) - number: Anvil library 100%, Anvil controller 37%, IronKV 84%, Atmosphere 83% - number: Specialized harness about 1.6x time and 2.2x cost on whole projects - number: 849 tasks from eight systems (nine projects): Anvil temporal library 104, Anvil vreplicaset controller 63, IronKV 118, mimalloc 89, Node Replication 29, NRKernel 204, Atmosphere 157, persistent storage 63, Vest 22 - limits: Failures center on discovering inductive invariants (facts that hold initially and stay true after every step). The Anvil library lemmas are not the same as proving that a new controller eventually reconciles. - how much was read: most of it read - source; notes: VeruSAGE: high overall rates hide controller-invariant difficulty; notes: VeruSAGE / VeruSAGE-Bench - record id: `verusage-2025` - VLTL-Bench (2025) - title: Verifiable Natural Language to Linear Temporal Logic Translation: A Benchmark Dataset and Evaluation Suite - what: It scores four steps of translating natural language to LTL. The steps are finding propositions, translating temporal structure (judged by logical equivalence, not string match), grounding to executable predicates, and classifying satisfying and violating traces. - number: datasets: Kitchen Assistant 10,000, Search-and-rescue 7,304, Traffic-light 7,319, Warehouse 7,457 - number: built from 36 expert nl2spec templates - number: Table 2: four generated datasets of 10,000; 7,304; 7,319; 7,457 entries (32,080 total) reusing 43 templates - number: 36 expert-crafted lifted NL-LTL pairs plus 7 new ones - limits: Its equivalence check only flags a difference. It does not say whether one formula is weaker or stronger. Two traces cannot establish equivalence or liveness. It has no proof, no code, and no audit of how far to trust agents. The text says three datasets but the table lists four. - how much was read: most of it read - source; notes: Temporal requirement translation: VLTL-Bench and GraFT; notes: papers - record id: `vltl-bench-2025` - Anvil (2024) - title: Anvil: Verifying Liveness of Cluster Management Controllers - what: Rust controllers are written as state machines. The authors prove they match an abstract model. They also prove that the model reaches the desired cluster state and then stays there, once the desired state stops changing. They call this eventual stable reconciliation. - number: ZooKeeper 1,134 executable / 8,352 model+proof lines; RabbitMQ 1,598 / 7,228; FluentBit 1,208 / 8,395 - number: total 3,940 executable, 23,975 proof/model, 2,326 trusted lines - number: parallel verification 154, 151, 96 seconds - number: about 2.5 person-months; first proof about two person-months, later ones about two person-weeks - limits: Trusts the temporal-logic embedding, the environment model, the specs of external APIs, Verus and the OS. Assumes faults eventually stop. The guarantee says nothing useful if the desired state changes forever. One bug came from an incomplete trusted spec of the ZooKeeper API. - how much was read: most of it read - source; notes: Anvil: Verifying Liveness of Cluster Management Controllers; notes: P06. Anvil: Verifying Liveness of Cluster Management Controllers - record id: `anvil-2024` - Autoformalizing Euclidean Geometry (2024) - what: Introduces LeanEuclid, 173 manually formalized Euclidean geometry statement/proof/diagram examples, and a Lean implementation of a variant of system E. A symbolic SMT engine fills implicit diagrammatic proof gaps and evaluates statement equivalence as E3. January2024 GPT-4/GPT-4V few-shot experiments report up to21.0% solver-proved statement equivalence and23.1% proof acceptance across the143-item evaluation set. SMT answers are admitted through an explicitly unsound axiom rather than reconstructed as Lean proof terms, so results require trusting the translation and SMT solvers as well as the Lean kernel. Separates explicit construction steps, chosen by humans/models, from non-construction diagrammatic/metric/transfer reasoning handled by SMT refutation. euclid_apply instantiates a rule, proves missing premises with SMT, and adds conclusions/witnesses; euclid_finish/assert discharge goals. E3 checks both implication directions plus contradiction detection; approximate comparison requires matching numbers/types of bound variables, selects top-n renamings by string similarity, and counts four directions of pre/postcondition obligations, adding preconditions to postcondition comparisons only after precondition equivalence. Dataset contains all48 Elements BookI propositions and125 UniGeo examples (25 each in five categories); texts are manually augmented/abridged. Appendices give a full axiom catalogue, a missing-case reconstruction for Elements24, formalization comparisons, exact prompts, partial semantic outcomes, and before/after repair scripts. - limits: Correctness is relative to the chosen geometric axioms and Lean-to-SMT encoding. AppendixC explicitly introduces an unsound axiom to admit successful SMT queries; there is no end-to-end kernel certification or SMT proof reconstruction. The reader did not audit or execute the released implementation. SystemE soundness/completeness is attributed to Avigad etal for its original axioms. This implementation adds construction, diagrammatic and transfer rules, strengthens one construction rule, and uses SMT real-number theories. The supplied paper does not mechanize a soundness proof of the modified implementation. The diagrammatic engine uses diagrammatic,metric,transfer rules. Equivalence checking additionally uses nine simple construction rules. Limited construction/instantiation makes E3 incomplete; failure to establish equivalence is not proof of inequivalence. Logical equivalence of closed true theorems can be weaker than faithful correspondence of hypotheses,objects and informal meaning. The paper itself reports a logically equivalent but unfaithful quadrilateral/parallelism example; success should not be read as comprehensive intent validation. Approximate equivalence is a bounded heuristic over variable unifications and clause counts, not a semantic distance with a correctness theorem. It excludes formulas with differing variable counts/types and depends on chosen clauses and names. Evaluation uses gpt-4-1106-preview and gpt-4-1106-vision-preview in January2024, one domain, manually prepared prompts and a small curated test split. No significance intervals or multiple-run variance are reported. Proof autoformalization receives a correct formal statement and informal proof, whereas statement autoformalization is evaluated separately. Proof success cannot establish an end-to-end informal-statement-to-correct-proof rate. Manual Elements repairs use ground-truth proofs and a minimum-change preference; Levenshtein ratios measure textual alteration, not time,semantic effort,independent automation success or proof validity. Data-contamination improbability is the authors inference from LeanEuclid being unpublished at experiment time. Models may have seen Elements/UniGeo or other formalizations; no pretraining audit is supplied. No distributed systems,Verus,Rust or TLA+ evaluation occurs. Reuse of domain rules and semantic equivalence checks in those settings is a reader hypothesis, not established transfer. - how much was read: most of it read - - record id: `autoformalizing-euclidean-geometry-2024` - CSP retrospective (2024) - title: The Challenges and Triumphs of CSP Based Formal Verification - what: Roscoe looks back on CSP, a notation for describing processes that run in parallel and talk to each other, and its model checker FDR. He covers a two-player-game encoding, a primary/secondary consensus handover protocol for blockchains with malicious nodes, a way to check deadlock freedom from the sum of component state spaces, and the Coco tool used by ASML. - number: Coco Platform is in use at ASML for photolithography machine software (as stated by the author, no figures) - number: the author's naive CSP blockchain model could not hold a realistic number of blocks, agents or keys (state explosion) - number: token-invariant deadlock check on a 3-component example: state space of 6 component states, one nontrivial invariant (t4) proves deadlock freedom - limits: Reflective paper written for CSP experts: no benchmarks or timings are given for the handover protocol or the local checking idea, and both are models checked in FDR, not code. The author lists proving the Bully algorithm and a distributed database protocol for all N as still-open challenges. - how much was read: most of it read - source; notes: The Challenges and Triumphs of CSP Based Formal Verification (2024) - record id: `csp-retrospective-2024` - Datadog Courier (2024) - title: How We Use Formal Modeling, Lightweight Simulations, and Chaos Testing to Design Reliable Distributed Systems (Datadog Courier) - what: Datadog modeled its Courier message queue in TLA+. It checked a liveness property named NoLostMsgs with TLC under weak fairness (an assumption that a continuously enabled step eventually runs). After a sequencer was added, TLC found a counterexample, and the design changed. - number: TLC scanned 5,515,710 distinct states with no violation in the first model - limits: It is a vendor post and not peer reviewed. The models do not validate the implementation. - how much was read: most of it read - source - record id: `datadog-courier-2024` - Datadog Courier blog (2024) - title: How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems - what: Datadog describes using TLA+ and TLC to model Courier, finding a message-loss counterexample after a design change, then combining model checking with simulation and chaos testing; it explicitly notes model/implementation mismatch limits. - limits: Already present in the local database; this pass reread and verified the page, so it is not newly added. - how much was read: most of it read - source - record id: `datadog-courier-blog-2024` - IronSpec (2024) - title: IronSpec: Increasing the Reliability of Formal Specifications - what: Tests whether a Dafny specification matches what developers expect. It uses three methods: checks with unconstrained inputs and outputs, small proofs of cases that should or should not be allowed, and mutations of the specification. - number: 14 specifications including six real verified systems; ten spec bugs confirmed by authors - number: 70 small test proofs, 21 needed help averaging 1.7 added lines - number: 61 surviving mutations, 13 judged intended - number: human effort hours to days; DaisyNFS about one to two hours - limits: The checks expose suspicious differences. They do not show the spec perfectly matches intent. It gives no success rate for autonomous spec generation. - how much was read: most of it read - source; notes: Specification errors and auditing; notes: IronSpec - record id: `ironspec-2024` - LVR (2024) - title: LVR: Mostly Automated Verification of Liveness Properties for Distributed Protocols with Ranking Functions - what: LVR turns a progress goal into ranking obligations and safety obligations: the measure decreases, stays nonnegative, cannot leave the pending condition without reaching the good state, and no deadlock occurs. It infers integer variables, bounds and polynomial coefficients. mypyvy and P-FOL-IC3 discharge the safety obligations. Humans supply any missing variables, bounds and tiers. - number: 37 out of 40 (§9.2, p.35:27) automatically; the rest manually in a few hours each - number: each final coefficient-solving query takes less than a second - number: eight models: ticket lock 68 lines, hybrid reliable broadcast 137, alternating bit 127, TLB shootdown 361, Paxos 133, multi-Paxos 135, stoppable Paxos 160, sliding window 119 - number: all eight protocols succeed: ticket lock, hybrid reliable broadcast, alternating bit, TLB shootdown, Paxos, Multi-Paxos, Stoppable Paxos, sliding window - limits: By default, scheduling assumes a globally bounded interval. This is a slightly stronger fairness than "infinitely often". For example, scheduling at times 1, 2, 4, 8 satisfies "infinitely often" but not any fixed bound. Prophecy variables and a three-tier ranking keep the original fairness. LVR's multi-Paxos leaves out the local scheduler and the I/O queue. So fewer lines than IronFleet does not mean equally less effort. A cheap solver query is not the whole cost. - how much was read: most of it read - source; notes: LVR and implicit ranking functions - record id: `lvr-2024` - miniCodeProps (2024) - title: miniCodeProps: a Minimal Benchmark for Proving Code Properties - what: Lean theorem statements about list, natural-number and sorting code, taken from TIPHaskell. It has a mode that writes the whole proof and a mode that searches over tactics. - number: 201 theorem statements: 86 easy, 115 medium/hard - number: pass@32 candidates - number: 201 Lean 4 theorem statements - number: GPT-4o 70/201 at pass@32 and 75/201 with refinement (Table 4) - limits: It has no spec generation or equivalence checking. It has no broad adversarial harness and no audit of axioms. Termination of recursion is not liveness of a distributed system. - how much was read: most of it read - source; notes: miniCodeProps - record id: `minicodeprops-2024` - Modelling the Raft Distributed Consensus Protocol in mCRL2 (2024) - what: # Whole read: Modelling the Raft Distributed Consensus Protocol in mCRL2 (authored by agents unless marked 🧑) Bora, Pham Duc Minh and Willemse, MARS 2024, EPTCS 399, pp. 7–20; DOI 10.4204/EPTCS.399.4; arXiv 2403.18916v1, submitted 27 March 2024. Read all 14 PDF pages, §§1–5, Figure 1, Table 1, acknowledgement and 14 references, in text and rendered pages. Also read the official MARS mCRL2 model, all nine property files, README and batch script statically. The downloaded paper matches the indexed PDF byte for byte. No author code, model checker, prover or solver was executed. Raw sources, hashes, scope and exact excerpts are retained separately. The paper contributes an action-based Raft model and modal µ-calculus formulas, with reported model-checking results for eight finite configurations. It does not establish unrestricted Raft correctness. More specifically, the authors write, “All main properties hold true for all configurations of Table 1; verification of each property takes roughly 2-3 times that of generating the state space” (§3, PDF p. 9 / printed p. 15; Q19). Static inspection also finds differences between the printed formulas, artifact formulas and model indexing. These narrow what a successful check can establish; they are not claimed reachable protocol counterexamples. ## Model and assumptions The model follows the TLA+ and LNT specifications and focuses on leader election and log replication: “Features such as cluster membership changes and log compaction have not been modelled” (§2, PDF p. 2 / printed p. 8; Q3). Node, Network and Client processes run in parallel; communicating actions and allow restrictions enforce synchronisation. Analysis requires a “fixed, concrete number of servers” (§2, PDF p. 3 / printed p. 9; Q4), rather than a proof quantified over all cluster sizes. One client sends a bounded sequence of unique natural command IDs, abstracting away command contents and application behavior (Q5). Client-to-cluster communication is synchronous; communication among servers is asynchronous through the network. The network holds a finite set of payloads, permits reordering and limits capacity (Q7). Thus two identical pending payloads collapse to one set element. Reliable mode supplies no scheduler fairness guarantee. Capacity can block sends; the broadcast admission guard uses the number of servers rather than the exact outgoing set size (Q47). These are model restrictions, not a general unbounded or duplicate-preserving network semantics. The scope description has a tension worth retaining. Section 2.1 says, “we do not analyse the Raft algorithm in the presence of message loss or duplication” (PDF p. 4 / printed p. 10; Q6). Table 1 nevertheless includes one lossy-and-crashing row, and §3 reports all main formulas true for every row (Q18–Q19). The model supplies an optional loss branch (Q8, Q47). The defensible reading is a primarily reliable model with one reported loss experiment; no duplication experiment is reported. Each server has role, term, log, commit index, vote state, replication indexes and a pending reply. Figure 1 adds a Crashed role to Follower/Candidate/Leader (Q9–Q10). Entries are Command(term, commandID), stored in zero-based mCRL2 lists; the authors explicitly warn that Raft’s original indexing is one-based (Q11). Only leaders accept client commands. Elections and heartbeats use nondeterministic actions; “we cannot analyse real-time requirements, nor the real-time performance of the algorithm” (§2.2, PDF p. 6 / printed p. 12; Q12). There is no sampling of randomized timeout intervals or elapsed-time state. A candidate votes for itself and sends vote requests to other nodes not already in its voter set (Q13). Stale RPCs are discarded. A received request computes and stores a reply before the later send, allowing a crash between the state update and reply transmission (Q14). The artifact’s resume transition preserves currentTerm, log and votedFor, but resets commitIndex, voterLog, nextIndex, matchIndex and the pending reply, and returns to Follower (Q46). Reply computation itself is atomic: the authors acknowledge it “does not accurately reflect real-world scenarios where the computation of a reply would involve multiple steps” (§5, PDF p. 13 / printed p. 19; Q35). Leaders send one log entry at a time; the model uses nextIndex backtracking to find a matching prefix (Q15). Majority replication commits only a current-term entry directly; earlier entries are committed indirectly with it (Q16). This is a protocol model without a refinement proof connecting a Rust implementation, application state machine or storage system to these transitions. ## What the properties check The action logic cannot inspect process parameters directly. The model therefore adds self-loop actions exposeLeader, exposeLogLeader and exposeLog (Q19, Q46). Safety formulas compare exposed states across arbitrary later execution, rather than merely reading two process parameters in one state. **Election Safety.** The printed formula forbids a different node from ever later exposing leadership in the same term (Q20); the artifact agrees (Q52). An additional existential formula asks whether a leader can be exposed at all; another asks whether two different node IDs can eventually expose leadership (Q21). The latter is reported true only with at least two terms. These are witness-existence checks, not eventual election on every execution. The least-fixed-point formula counts possible leader announcements whose ID differs from the most recently stored ID, starting with id=1 and count=0 (Q22, Q53). It does not compare the associated terms or demand globally distinct IDs; a later reuse of an earlier ID can count. It is therefore more precise to describe its actual counter than to call it a proof of MaxTerm distinct leaders or guaranteed succession. **Log Matching.** The intended statement is that the same index and term imply identical prefixes through that index. The paper’s guard instead tests complete entry equality, log1.index == log2.index, which is a stronger antecedent and makes the standalone requirement weaker (Q23). The artifact repairs this to term(log1.index) == term(log2.index) and checks both list bounds (Q49). Both versions compare slice(log,1,index+1). The supplied slice helper begins at zero and excludes its finishing index (Q42). It therefore omits position 0, the first actual entry; initial logs contain no dummy element (Q48). Reader-created one-entry logs [Command(1,1)] and [Command(1,2)] have matching terms at position 0, but both compared slices are empty. This separates the intended first-entry equality requirement from the artifact formula. It does not establish that these two logs coexist in a reachable model state. **Leader Completeness.** After advanceCommitIndex(old,new,term,log), the formula asks that selected entries occur somewhere in every later-term leader’s exposed log (Q24–Q25, Q51). The membership consequent is weaker than retaining each entry at its original index. There is also an indexing discrepancy: it selects log1.index for old < index <= new, while the model’s getTerm translates Raft index to list position index-1 (Q41). With a singleton log and new=1, the formula selects zero-based position 1 rather than 0. No semantic result for out-of-bounds list selection, checker failure or reachable counterexample is asserted here. For commits in the sole elected term of a one-term configuration, there is no later leader term to test; the auxiliary leader/commit existence checks do not make every main formula nonvacuous. The event itself has a separate static inconsistency. On the successful AppendEntriesResponse branch, advanceCommitIndex computes its new index from the old matchIndex array, whereas the following Node transition computes commitIndex after incorporating the new acknowledgement (Q45). A local valuation with one entry, old matchIndex=[0,0,0] and an acknowledgement of entry 1 from another server yields event new index 0 but state commit index 1 using the supplied count helper (Q43–Q45). Reachability of that valuation was not established. The source expression mismatch is sufficient reason not to equate the event range automatically with all newly committed entries. **State Machine Safety.** The intended statement forbids two nodes applying different entries at the same committed index. Paper and artifact compare slice(log1,1,commitIndex1) and the corresponding slice of a later other node with commitIndex2 >= commitIndex1 > 0 (Q26, Q50). Under the artifact helper, position 0 is excluded. For commitIndex1=1 both slices are empty, independent of the first entries. Thus a successful check of this formula alone cannot establish agreement on the first committed entry. Again, this is formula coverage established by symbolic values, not a reachable protocol failure. An existential companion asks whether two different nodes can expose positive commit indexes in sequence, the later one at least as large (Q27). It does not repair prefix coverage or prove every request eventually commits. Append Entries is omitted as a separate checked property because the authors consider it immediate from the modeled leader operations (Q19). ## Reported evaluation and concrete cases The authors used manual interactive simulation and safety formulas throughout development to find model bugs (Q19). Table 1 gives eight configurations, all with three nodes and capacity three. The following preserves the reported rounded sizes and approximate generation times; every row is retained in numeric-evidence.json (Q18, §2.2, PDF p. 8 / printed p. 14): - 2 commands, 1 term, reliable, no crashes: 2.14×10^5 states, about 1 minute - 1 command, 2 terms, reliable, no crashes: 1.17×10^6 states, about 2 minutes - 1 command, 3 terms, reliable, no crashes: 1.32×10^7 states, about 13 minutes - 2 commands, 1 term, reliable, crashes: 1.79×10^7 states, about 5 minutes - 2 commands, 2 terms, reliable, no crashes: 2.25×10^7 states, about 19 minutes - 2 commands, 1 term, lossy, crashes: 5.97×10^7 states, about 105 minutes - 1 command, 2 terms, reliable, crashes: 1.48×10^8 states, about 24 minutes - 1 command, 3 terms, reliable, crashes: 2.38×10^9 states, about 820 minutes The first seven times use a 2017 Macbook Pro with 16GB; the last uses an unspecified compute server and is explicitly “not directly comparable to the other runtimes reported” (Table 1; Q18). No repetitions, error bars, exact CPU, per-formula times or implementation latency/throughput are supplied. The paper reports roughly 2–3× generation time for each main property using symbolic verification (Q19, technique reference Q38). The base configuration compresses by almost factor 5 under strong bisimilarity; no exact reduced size is given (Q17). The artifact README names mCRL2 release 202307.1 and gives explicit and symbolic commands (Q56). Its default model has five servers, two requests, one term, capacity five, no loss/crashes, and PRIORITISE=false (Q40). Seven individually guarded Node instances support the small configured sizes (Q48). The batch script substitutes three servers/capacity three, requests 1 or 2, terms 1–3, and both Boolean loss/crash settings, for 24 combinations (Q57). It includes all .mcf files, including timeout/client-command checks (Q54–Q55); neither the script nor their inclusion is a saved result log. Do not infer that every batch combination or every extra formula was successfully checked. The official archive contains source and instructions, not experiment output or an identified experiment commit. MAJORITY is ceil(NumberOfServers/2) in the artifact (Q40). That is a strict majority for the odd sizes three and five here, but only half for even sizes. No arbitrary-size correctness claim should be inferred from the configurable constant. Section 4 illustrates how modeling style changes generated state count. A Boolean process with conditionals after the action yields 3 states/6 transitions; moving the choice before the action yields 1 state/2 transitions, strongly bisimilar to the first (Q29–Q30). The tool can then remove an unused process parameter. The example supports a concrete state-generation lesson; it is not a measured uniform threefold Raft reduction. The claim that an early inexperienced model was “orders of magnitude larger than needed” is an author observation without a controlled Raft benchmark (Q29). ## Relation to the research topic The direct contribution is a finite distributed-protocol model and action-logic encodings of selected Raft requirements, with bounded symbolic checking and concrete resource measurements. The authors describe the modal formalizations as apparently new while acknowledging the prior TLA+, LNT and Verdi work (Q2, Q33, Q39). This is useful evidence about specification design, state exposure, state-space growth, crash abstraction and model/property correspondence. The TLA+ comparison is specific. The authors say TLA+ answers stale requests but drops stale responses; their model follows LNT in discarding both. They follow TLA+ when bounding transmitted commit information, and add reboot behavior they say is absent from the LNT model (Q31). These are attributed source discussions, not an independent audit of all upstream artifacts. Section 4’s view that imperative LNT is more appealing to average engineers is an author opinion, not a user study (Q32). For evaluating LLM agents writing specifications or proofs, the useful checks are reader inferences: validate a formula against its helper definitions, include first/last-index examples, inspect whether instrumentation exposes the actual state transition, distinguish term equality from entry equality, and separate existential witnesses from inevitable progress. This paper supplies concrete reasons these checks matter, but it contains no LLM-agent, Verus, verified-Rust or implementation-refinement evaluation. The liveness boundary is explicit: “Stronger liveness requirements, asserting that always inevitably a leader must be elected, are simply not true in our model” (§5, PDF p. 13 / printed p. 19; Q34), owing to crashes/loss and bounded terms. More complex configurations, stronger liveness, finer reply computation and real-time verification remain future work (Q33–Q36). Successful bounded checks therefore support the specified finite model and literal formulas, subject to the correspondence issues above. They do not independently establish the stronger prose requirements. ## Evidence and review quotes.json contains 57 literal excerpts with file, section, PDF/printed page where applicable, line ranges and character offsets. combined.quote-source.txt contains every excerpt for database validation. source-files.json retains raw hashes; source-scope.json distinguishes whole-paper and static artifact coverage. numeric-evidence.json preserves all table rows, and static-witnesses.json records reader-created separating values. reader-diagnostics.json separates source claims from reader findings. REVIEW.md and validation-final.json track consultation, independent review and mechanical checks. - limits: Fixed finite cluster,client commands,terms and finite-set network capacity; one synchronous client and asynchronous interserver RPC; no membership changes/log compaction; nondeterministic untimed timeouts,atomic reply computation,optional crashes/reboot/loss. Eight reported three-node configurations. No duplication/multiplicity or fairness guarantee. Static indexing/formula/event mismatches qualify property coverage. - how much was read: most of it read - source - record id: `modelling-the-raft-distributed-2024` - mypyvy repository (2024) - title: mypyvy: A Research Platform for Verification of Transition Systems in First-Order Logic. - what: mypyvy is a symbolic transition-system language inspired by Ivy. Its README documents inductive invariant checking and inference and gives a lock-server example. - number: 43 top-level examples, 83 more under misc and fbii - number: examples/misc/pd at the inspected checkout has 39 .pyv files covering cache, consensus, lock service, ring election, sharded KV, TLB and Paxos variants - limits: The README gives setup and examples, not benchmark results. - how much was read: most of it read - source - record id: `mypyvy-repository-2024` - RTLola UAV Monitoring (2024) - title: Monitoring Unmanned Aircraft: Specification, Integration, and Lessons-Learned. - what: Reports integration of RTLola-generated runtime monitors for electric unmanned aircraft across log analysis, hardware/software-in-the-loop testing and flight tests, with an abstraction decoupling stable specifications from environment-specific integration. Defines translation layers between system outputs and monitor inputs, validates event-source mappings, and presents flight-phase detection, detect-and-avoid, geofence, sensor/system checks and other representative specifications. The monitor was integrated into development environments and tested in a flight with feedback to ground control. - number: Three organizations contribute authors: CISPA, Volocopter and DLR - number: A large case study covers representative specifications across multiple integration environments - number: A final test flight reports monitor feedback to the remote pilot - number: No quantitative accuracy/runtime benchmark table is reported - limits: Some specifications are unpublished because they require complete system knowledge. The monitoring approach checks observed streams and integration assumptions; external components must conform to their specifications. The paper reports a large case study and test flight rather than controlled detection-accuracy or runtime benchmarks. - how much was read: most of it read - source - record id: `rtlola-uav-monitoring-2024` - SCTP formal analysis (2024) - title: A Formal Analysis of SCTP: Attack Synthesis and Patch Verification - what: Humans build a Promela model of SCTP from the RFCs. They choose ten LTL properties and four attacker models. SPIN verifies the properties and KORG synthesizes attacks. This shows that the RFC 9260 patch removes the modeled CVE-2021-3772 attack. - number: ten LTL properties; four attacker models - number: long Off-Path checks 123:42 and 86:10 (mm:ss) unpatched vs 7:07 and 6:48 patched, on a 16GB M1 MacBook Air - number: KORG changes: 80 existing and 213 new lines, plus 102 shell lines - number: Ten LTL properties verified without attacker; long off-path checks 123:42 and 86:10 (mm:ss) unpatched vs 7:07 and 6:48 patched - limits: Two peers and buffers of size one. Excludes crashes, restarts, congestion control and most data transfer. The attacks are real only against the model. No theorem that the code matches the model. Model size and modeling time are not stated. - how much was read: most of it read - source; notes: SCTP formal analysis: humans supply the model, properties and attacker; notes: SCTP formal analysis: humans supply the model, properties and attacker - record id: `sctp-formal-analysis-2024` - Selene (2024) - title: Selene: Pioneering Automated Proof in Software Verification - what: It isolates existing Isabelle lemmas from seL4, supplies their dependencies and examples, and retries GPT-3.5/GPT-4 proofs with error repair and retrieval. - number: Table 2 p. 1781: GPT-4 five-attempt rates 51.8, 12.5, 1.7, 15.8 percent across short, medium, long and proof-with-definition categories - limits: It covers selected kernel lemmas only. Hints from the proof prefix are privileged information. - how much was read: most of it read - source; notes: seL4: from whole-proof guessing to proof-state search - record id: `selene-2024` - VeriSMo (2024) - title: VeriSMo: A Verified Security Module for Confidential VMs - what: VeriSMo is a Rust security module at AMD SEV-SNP VM privilege level 0, verified with Verus. It protects its own data from an untrusted hypervisor and guest OS while providing guest page management, code integrity, runtime attestation and secret management. A machine model proves security under external interference; tracked permissions and locks verify the implementation under internal CPU concurrency. Security types constrain explicit and implicit information flow. Records exact successful-read and reachability conditions of security theorems, five preservation lemmas, initial page-table/permission assumptions, trusted hardware and byte-casting boundaries, shared-memory restrictions, register rules and declassification. Reconstructs the remapped-secret-page attack and zeroing fix, all API latency/LOC figures, baseline feature mismatch and the restricted-interrupt Linux/Hyper-V dependency. Distinguishes paper-reported proofs and fixes from independently replayed verification or exploits. - number: about 2:1 annotation/code ratio - number: about six minutes verification on 32 cores - number: 31 lines of unsafe Rust for hardware primitives - limits: §§2.1,3.1–3.2: Trusts CPU/firmware security behavior while hypervisor and guest OS may be malicious. Physical attacks are excluded and denial of service remains possible: the hypervisor can stop the host or refuse vCPU scheduling. This is a conditional safety result, not availability, scheduling fairness or distributed liveness. §§2.1,3.2: VeriSMo runs at VMPL0; guest at VMPL3, though VMPL1/2 could be used. Both use restricted interrupts, so the hypervisor can inject only #HV and a doorbell supplies the requested event. Mainstream Linux v6.8 lacked this support in KVM/guest, so experiments use Hyper-V and a modified Linux guest. The proof does not establish safety for an arbitrary unmodified VM/interrupt configuration. §§2.1,5.1: Hardware model includes CPU registers, memory, guest and nested mappings, RMP and cache/TLB-related state. E0 is VeriSMo, E3 guest, Ehv hypervisor. Guest and VeriSMo share an encryption key; VMPL access permissions separate them. Sibling VMs are omitted because the model grants the hypervisor a superset of their capabilities. Reads of externally shared memory/registers are unconstrained. §§4.2,5.1.2: External interference is handled in a machine state transition model; internal concurrency uses ownership and permissions. Primitive instruction wrappers contain trusted unsafe assembly and fully trusted postconditions derived from AMD manuals. Only critical memory/cache operations are modeled, under stated restrictions. Direct GHCB communication replaces code that might trigger #VC; unexpected #VC or other interrupts terminate the VM. Hardware-model completeness and faithfulness remain assumptions. §5.2 Listings 4–7: Confidentiality assumes an invariant-valid state, a successful secret E0 read, equal system physical addresses and a successful other-entity read; it concludes that the latter is encrypted. Integrity assumes invariant validity, attack reachability and successful E0 reads in both states; it concludes equal returned data. Neither theorem forces a read to succeed. Invariant preservation restricts E0 operations by software preconditions; attack-model closure covers arbitrary non-E0 operations. These are model-level security statements, not a complete cryptographic or microarchitectural side-channel theorem. §5.3 Lemma 1: After a validated VM-private GPA binds to an SPA, hypervisor changes leave it bound to the same SPA or make access fail, provided E0 does not revalidate an already bound GPA. Lemma 2: VM-private secrets remain encrypted to hypervisor reads; the guest C bit is required, whereas a permanently set RMP validation bit is unnecessary and cannot be assumed. A hypervisor can invalidate memory without gaining plaintext access. §5.3 Lemma 3: VMPL0-private memory remains unreadable by E3 and its E0 read either returns the original value or fails under Ehv/E3 actions. rmpadjust must not grant guest access to private pages. Lemma 4 extends same-or-fail translation from GPA to GVA using protected page tables. Lemma 5 requires a one-to-one mapping and separate page-table versus ordinary write APIs. §§5.3.3,6.1.1: Initial E0 page tables are trusted to reside in designated private PT memory and map GVAs one-to-one to GPAs. pt_write must preserve both conditions. Initial uniqueness of memory permissions is also assumed, then ownership/non-copyability preserves uniqueness. These initialization assumptions must be checked in a deployment; preservation alone does not establish the initial state. §§5.3.4,6.1: The connecting proof turns software-tracked constraints into hardware constraints when an operation succeeds. SnpPointsTo records GVA identity, optional value and software/hardware RMP/page-table/is_pt attributes. Tracked resources cannot be copied or freely constructed. Fine-grained permissions avoid carrying a global machine state through each implementation proof. §6.1.1: Object sizes, object/byte casts with consistent size and unique bindings, and raw-memory splitting/merging with consistent attributes are trusted specifications or axioms. Rust ownership does not verify these representation facts automatically. Listings 8–11 illustrate permission interfaces and deliberate rejected operations; their abbreviated syntax is not a separately compiled proof artifact. §6.1.2–6.1.3: Lock acquisition transfers a stored memory permission to the holder; release requires restoration of its invariant and returns ownership. Values can change between separate acquisitions. A local lock cannot prevent malicious guest/hypervisor access to a shared page: its permission allows copying and writing but no borrowed reference. Unrestricted reference access requires copying to private memory or safely making the page private. §6.1.4: Registers have per-CPU permissions. GHCB MSR is shared and reads are unconstrained. Writing IDTR/GDTR transfers ownership of the pointed-to memory to hardware. Explicit RIP/SP access is excluded. Updating CR3 requires mutable permissions for the page table and all usable virtual memory, so the single kernel page table is installed only once at boot. These restrictions limit the verified implementation, rather than verifying arbitrary register-manipulating code. §6.2: Security types pair a value with per-entity possible-value sets; full set means secret and singleton means public. Operators propagate sets; comparisons require public operands, and memory addresses must be public. Non-confidential memory may contain only values public to the relevant entity. Trusted downgrade converts a set to a singleton; downgraded keys fail crypto preconditions requiring full secrecy. §6.2 reader qualification: A possible-value set is not a probability distribution or an entropy bound. The paper describes a tracking discipline and trusted cryptographic interfaces, not a general proof that arbitrary full-support values are uniformly random. Binary-operator equation is printed with a op b inside a set quantified over v/u; it is illustrative notation and is not a checked artifact definition in this reading. No flaw in the external implementation is inferred. §§3.4,6.2,7.1: Only three secret categories are used: hardware VM communication keys, VeriSMo key pair and guest symmetric keys. Their uses go through a completely trusted external verified crypto library. Trust also includes ground-truth axioms, functional/security specifications, primitive APIs, Rust core interfaces, Verus and Rust compiler. External HACL verification is not re-established by this paper. §3.4: Guest page sharing must use the module API; otherwise those pages will not be validated again. LockKernel is callable once and removes guest write permission from declared kernel-code pages and supervisor-execution permission from other pages. The caller chooses code ranges; this does not establish that the chosen code is benign or that a malicious guest supplies a truthful semantic runtime measurement. §3.4: ExtendPCR maintains a hash chain beginning with starting code/configuration; Attest binds a nonce, a hardware-attested module identity and module-attested chain. DeriveKey ties a retained encryption key to the runtime measurement and guest uses Encrypt/Decrypt without receiving it. Protocol/crypto behavior and meaning of measured guest inputs depend on trusted specifications; no remote distributed attestation workflow is evaluated. §7.2: OSMEM executable ranges and tracked permission map share an invariant equating range membership with possession of page permissions. One lock protects both the bookkeeping and permission-controlled RMP modifications. Handler range checks are forced by the need to obtain a tracked page permission. This establishes specific safety/range constraints, not every possible API intent property without a specification. §8.1: API averages/standard deviations come from 100 calls each. Switch measures the full VMPL3→hypervisor→VMPL0→hypervisor→VMPL3 round trip. Attestation has much greater variability than cheap calls; LockKernel is expensive but once per session. Table 1 provides milliseconds and thousands of cycles, not proof runtime. §8.2: UnixBench compares VeriSMo, Linux-sm, C-sm and Linux. Authors report almost zero overhead and comparable performance, but do not publish per-bar confidence intervals or an all-workload equivalence bound. Figure 5 contains visible differences, including Linux above modules on some bars. Linux-sm/C-sm implement only WakeUpAP, SetPageShared and SetPagePrivate, whereas VeriSMo provides more APIs. These comparisons are not a controlled isolation of verification cost for identical feature sets. §§8.2,8.4: Up to 40% Hecate slowdown is attributed to prior work and its #VC trapping, not a new head-to-head run. AMD SVSM/COCONUT performance is not compared because the hypervisor differs. Unsafe block counts (32 VeriSMo, 150 AMD SVSM, 235 COCONUT) are source snapshots reported in 2024; lower block count alone proves neither comparative security nor absence of bugs. §8.3: Reports about six minutes for both proof layers using multithreaded Verus and Z3 4.11.2 on 32 cores. Table 2 includes 803 Verus-library axiom LOC and 101 trusted LOC as well as project axioms/trusted wrappers; the 31 unsafe Rust LOC for hardware instructions is not the total trusted code. The allocator interface is trusted, although its implementation is verified. Proof/spec annotations are approximately twice executable implementation size, not a uniform ratio for every module. §8.4.1: The remapping vulnerability was detected during verification; authors report an AMD disclosure in early May 2023 and security fix, with early COCONUT still retaining the older bug. The reading checks the paper account and cited commit pointer only, not a reproduced exploit, current project state, or independent fix audit. No artifact or proof replay was executed. - how much was read: most of it read - source; notes: VeriSMo: A Verified Security Module for Confidential VMs - record id: `verismo-2024` - Verus SOSP 2024 (2024) - title: Verus: A Practical Foundation for Systems Verification - what: A Rust verifier for systems code. In its EPR-enabled method (EPR is a restricted logic that the solver can decide), the developer writes the inductive invariant (a fact that holds initially and is preserved by every step) and the tool checks it automatically. The evaluation (Section 6, Figure 9) is at the component level. - number: 6.1K lines of implementation and 31K lines of proof - number: IronKV 1,533 exec / 4,509 proof / 1,613 trusted lines; verifies in 18 s on eight cores vs Dafny 201 s - number: node replication verifies in 9 s vs 228 s - limits: It applies to the paper's EPR-enabled method, not to finding arbitrary invariants. - how much was read: most of it read - source; notes: S30; notes: Verus: A Practical Foundation for Systems Verification - record id: `verus-sosp-2024-2024` - ByMC (2023) - title: ByMC: Byzantine Model Checker, and Survey on Parameterized Verification with Threshold Automata - what: ByMC models nodes that count messages against thresholds, abstracts the counts, and uses an SMT solver to check safety and liveness for every allowed number of processes and faults. Verified for both safety and liveness: folklore and Srikanth-Toueg reliable broadcast, asynchronous Byzantine agreement, nonblocking atomic commit, condition-based consensus, CF1S, C1CS, and BOSCO. The randomised protocols (Ben-Or, Byzantine consensus, k-set agreement and RS-BOSCO) get safety plus almost-sure termination (they finish with probability 1). - number: Table 4 (§6.2, p.5:20) lists the verified protocols - limits: The randomised cases hold only against adversaries that respect round boundaries. The Tendermint one-round agreement analysis covers safety only. The benchmarks are compact models, not runnable implementations. The review notes did not recover the original 2018 ISoLA chapter and read the 2023 survey instead. The review notes found no human person-time. - how much was read: most of it read - source; notes: ByMC: fully parameterized model checking; notes: Ivy, ranking, Coq, and Iris case index - record id: `bymc-2023` - Decouchant-KulahciogluOzkan-Zhou liveness testing (2023) - title: Liveness Checking of the HotStuff Protocol Family - what: Extends Twins automated BFT testing with temperature-based and lasso-based liveness detection using partial/hot state abstractions, and compares them with bounded-time liveness checks on HotStuff, 2-Phase HotStuff, and Sync HotStuff. Defines partial states that retain liveness-relevant view/lock information and hot states used by temperature and cycle detection; lasso detection caches repeated hot states without controlled scheduling. Twins scenarios use two partitions, four honest processes and one Byzantine twin, with 1,000 unit tests for execution timing and 10,000 random scenarios per protocol evaluation. Baseline timeout checks report network-partition delays as violations; temperature/lasso distinguish persistent conflicting locks. Sync HotStuff’s early version exposes both safety and liveness faults. - number: 1,000 timing scenarios per configuration - number: 10,000 random scenarios - number: HotStuff small-timeout 20-round 97.6% false positives - number: 2-Phase 20-round temperature 0.17–1.92% liveness, 0 false positives - limits: Testing is bounded by generated rounds and state abstraction quality; temperature threshold selection trades false positives against runtime, while lasso can use more memory. Twins models Byzantine behavior through twin replicas and injected network faults; results are not formal proofs and depend on implementation/version (Sync HotStuff eprint 20190312:115828, Δ=50 ms). - how much was read: most of it read - source - record id: `decouchant-kulahciogluozkan-zhou-liveness-testing-2023` - FuzzBtor2 (2023) - title: FuzzBtor2: A Random Generator of Word-Level Model Checking Problems in Btor2 Format - what: Provides a self-contained C++11 fuzzer that recursively generates random QF_ABV transition systems and safety properties as Btor2 files, with controls for state/input counts, arrays, constraints, bad properties, tree depth, and candidate widths. Tests expose bugs in AVR and Pono. - number: 200 calls; 100 BV and 98 ABV cases, 2 ABV generation failures - number: 22–58 lines, 39.2-line average; all passed catbtor - number: AVR: 16 safe, 24 unsafe, 22 uniquely solved, 157 errors, 1 timeout - number: Pono: 44 safe, 27 unsafe, 53 uniquely solved, 127 errors, 0 timeouts - limits: The target is Btor2's quantifier-free bit-vector-with-arrays theory; users choose a set/range of widths rather than an exact sort. Random generation is probabilistic and may fail for some parameters. Output/fair/justice keywords and liveness/fairness are unsupported. The evaluation uses small 22–58-line files and one-hour checker timeouts, establishing bug-finding value rather than broad performance. - how much was read: most of it read - - record id: `fuzzbtor2-2023` - Grove (2023) - title: Grove: a Separation-Logic Library for Verifying Distributed Systems - what: Grove is a separation-logic library for verifying distributed systems, implemented as an extension living in the Perennial repository; its README points to a VRSM case study and Go source. - number: 12x lines of proof as lines of code (§6) - number: 67-73% of single-core Redis throughput; about 2x throughput from one to three servers - number: 1,597 lines of added Coq reasoning principles - number: proof-to-code ratio around 12 - limits: README points to an SOSP paper for details; this record covers repository documentation only. - how much was read: most of it read - source; notes: papers and boundaries - record id: `grove-2023` - Hydra (2023) - title: Hydra: Effective Runtime Network Verification - what: It compiles monitor programs that operators write in Indus into P4 code. The P4 code checks each packet at every switch hop and at the last hop. - number: 11 properties in 14-64 Indus lines generating 102-211 P4 lines - number: Aether baseline 44.53% of header-vector capacity, one checker raises it to 52.14% (7.61 percentage points) - number: compiler about 2,500 OCaml lines - number: known Aether bug reproduced and confirmed - limits: The expressiveness theorem is about the language, not the network. There is no liveness. Humans write the monitors, the field mappings and the linking. The testbed could not saturate switch throughput. - how much was read: most of it read - source; notes: Hydra: Effective Runtime Network Verification; notes: Hydra: Effective Runtime Network Verification - record id: `hydra-2023` - Message Chains / UPVerifier (2023) - title: Message Chains for Distributed System Verification - what: Records the chain of messages that caused a later message, so causal facts can be stated directly without inventing ghost state (extra proof-only variables). It is used to check proofs of P-like programs for any system size. - number: 19 evaluated cases: 16 reused mypyvy models plus Tor/onion routing, GlobalClock (150 lines, under 2 s) and 2PC-MVCC (250 lines, about 12 minutes) - number: Pages 2224-2250 - limits: It automates the bookkeeping for proofs that are supplied, and mines facts from examples. It does not discover invariants in general. The review notes did not locate a complete fixed archive of all 19 models. - how much was read: most of it read - source; notes: Message Chains / UPVerifier: full 19-case paper set recovered; notes: Message Chains for Distributed System Verification - record id: `message-chains-upverifier-2023` - Netrix guided consensus testing (2023) - title: A Domain Specific Language for Testing Consensus Implementations - what: Presents Netrix, a central networking server plus DSL for programmer-guided exploration of consensus implementations. Filters control message drops, delays, modifications and reordering; monitor state machines express safety/bounded-liveness assertions; residual nondeterminism is explored by PCTCP and tests are replayable. Formalizes protocol, monitor and product transition systems and proves monitor expressivity for complete runs. Instrumentation shims intercept RPC/message APIs; filters and key-value context encode semantic waypoints. Case studies cover Tendermint, Raft and BFTSmart across Go and Java, including regression tests and protocol-coverage measures. Filter-distance analysis predicts when PCTCP alone can find reorderings. - number: 34 unit tests total: 15 Tendermint, Raft tests, 8 BFTSmart (§5) - number: Instrumentation: 600 LOC added to 150k LOC Tendermint; 120/16k Raft; 150/16k BFTSmart - number: Tendermint found 4 new deviations; one liveness bug increased consensus time 30s→6min; fixes acknowledged - number: Raft tests found 4 documented bugs; PCTCP alone found none of new Tendermint or known Raft bugs - limits: Requires instrumenting communication APIs and serialization adapters; tests explore bounded executions (e.g., MaxEvents=1000, depth=10, 30s iterations), not exhaustive correctness. PCTCP cannot introduce Byzantine behavior and rapidly loses probability on deep reorderings. Findings are implementation-specific scenarios; passing tests do not prove safety or liveness. BFTSmart case found no bugs, possibly reflecting prior testing maturity. - how much was read: most of it read - source; notes: Netrix (2023) - record id: `netrix-guided-consensus-testing-2023` - PRISM proof repair infrastructure (2023) - title: Proof Repair Infrastructure for Supervised Models: Building a Large Proof Repair Dataset - what: Introduces the initial PRISM Coq proof-repair dataset and its extraction infrastructure, with roughly 200 unique changes at publication. Defines proof repair with known repaired theorem statements and kernel-checking metrics; explains build, serialization, command alignment and project-history obstacles. It proposes infrastructure and evaluation practices, not a trained proof-repair model. Inputs include old theorem/proof, new theorem, changed dependencies, project state, raw text/ASTs/identifiers and intermediate goals/hypotheses. Partially applying an aligned pair of Git commits while leaving out one proof edit produces a broken proof plus supervised repair target. Across-project and chronological splits test different generalizations but no fixed split is committed. SwiM selects/clones cached opam environments; bwrap remaps cloned switches because binaries hard-code paths. strace recovers I/Q/R dependency flags from actual builds. SerAPI supplies states, while a heuristic parser and Vernacular queries work around extensible syntax, unqualified names, missing definition/conjecture queries, nested proofs and version drift. Command alignment solves a thresholded normalized-Levenshtein bipartite assignment over changed commands; it ignores proof bodies when comparing command text, supports moves across files/reordering, and uses t=0.4 to reject poor matches. Supplementary material lists target/candidate projects and a concrete hoare-tut exgcd.v repair replacing cutrewrite with replace, showing compact commit/diff/environment encoding of a much larger state. - number: initial release: a few Coq projects, roughly 200 unique changes - number: main text expansion target: 60 opam projects; appendix lists an additional 224 candidate projects - number: supplement Table 1 reports 171,775 proofs and 2,984,471 sentences; Table 2 reports 335,223 proofs and 6,078,063 sentences, each approximate one-snapshot inventory rather than repair counts - number: build attempts: about 68% failure across Coq versions 8.9–8.15 (seven major versions) - limits: The primary task assumes the new theorem and dependencies have already been repaired; successful checking establishes that stated theorem, not that a changed specification captures intent. Specification/definition repair lacks a satisfactory metric; exact equality is conservative and BLEU-like distances are inadequate. Initial release has a few projects and roughly 200 changes, whereas 60-project expansion and later large-scale coverage are planned. Main text and supplementary Table 1 use inconsistent wording about whether all 60 projects are already included; appendix prose explicitly calls repositories current and pending. Supplementary proof/sentence totals describe approximations from one snapshot per repository and are not repair-instance counts. Name resolution relies on handcrafted scopes and assumptions about shadowing; conjecture tracking assumes Show Conjectures order and proof-boundary behavior; grammar extensions can defeat serialization. Roughly 68% build-failure rate counts attempts across seven Coq versions 8.9–8.15, so it is not a rate of irreparably broken projects. No trained ML model, proof-repair accuracy, fixed benchmark split, or model comparison is reported. - how much was read: most of it read - source - record id: `prism-proof-repair-infrastructure-2023` - Real-Time Double-Ended Queue Verified (Proof Pearl) (2023) - what: Isabelle/HOL verification of a persistent purely functional deque based on Chuang and Goldberg. Refines its representation into explicit rebalancing phases and proves list-based functional behavior and invariant preservation. Fixed work on each update yields worst-case constant-time operations under a unit-cost arithmetic/list-primitive model; paper gives a code-inspection complexity argument, not timing benchmarks. The main work is specifying intermediate rebalancing states: two simultaneous abstractions describe current observable contents and the eventual balanced ends; a remaining-step function and size_ok inequalities ensure both ends survive updates before rebalancing finishes. Split stack representations avoid final linear append. Small bounded deques are expressed with constructors rather than an implicit list-length restriction. - limits: Sequential immutable structure with persistence, not a concurrent deque. Interleaved enqueue/dequeue operations occur BETWEEN bounded rebalancing slices, not simultaneous threads. Prior versions remain available through functional values. Correctness equations require invar, except the stated empty abstraction equation. first/dequeue behavior has a nonempty-list precondition. Missing empty/invalid-state pattern cases are outside that specified behavior; no total checked error-return API is described. The abstract meaning is listL, with listR=rev(listL). Left/right enqueue prepends in the corresponding orientation; valid nonempty dequeue removes the first corresponding element. Isabelle HOL lists/naturals and total logic do not imply defined useful behavior for arbitrary malformed constructors. Rebalancing has phase-dependent invariants. A local Small1 state cannot independently determine its final list because transferred big-side elements are missing; the joint States abstraction supplies that information. Specifications over the partial local abstraction must stay in phase-appropriate contexts. Runtime code is the yellow-background material. list/list_current, remaining_steps and proof invariants may traverse large lists; these are ghost definitions for verification, not functions called by every deque update. Similarly size_new and some overloaded size meanings are logical descriptions; executable counters implement needed constant-time size tests. Worst-case O(1) uses constant-time arithmetic, list cons/head/tail and fixed-depth nonrecursive helper calls. It is not a measured physical real-time deadline or a claim about unbounded-integer bit complexity, garbage collection, code-generator/runtime correctness or allocator latency. §5.7 argues constant time from nonrecursion and primitive operations. The paper presents no separate instrumented cost semantics/theorem, exact instruction bound, executable benchmark, space-allocation measurement or proof-effort comparison. Public updates on a valid Rebal state perform four joint steps; an update starting rebalancing performs six. Phase transitions and invariant slack, not merely the 3:1 idle balance condition, establish safe completion before an end can run out. The publication says the formalization has 4,400 definition/proof lines and changes some names for presentation. Several printed code snippets have type/name/direction inconsistencies, so the displayed pseudocode should not be copied as an executable artifact without consulting the formal source. This record has not checked the repository. - how much was read: most of it read - source - record id: `real-time-double-ended-2023` - The FormAI Dataset: Generative AI in Software Security through the Lens of Formal Verification. (2023) - what: Presents FormAI: 112,000 GPT-3.5-turbo-generated compilable C programs, labeled using ESBMC bounded model checking. Authors assert counterexample-backed findings eliminate false positives; absence of findings is bounded by verification settings. Dynamic zero-shot prompts combine 200 task types and 100 styles. ESBMC runs with –unwind 1 –overflow –multi-property –timeout 30; labels are retained only for runs finishing within 30 seconds. Distinguishes verification successful, unknown and failed; maps detections to 42 CWEs. Reports seven ESBMC bugs found and subsequently fixed. - number: 112,000 compilable C programs totaling 8,848,765 lines of code (average 79 LOC, mode 47 LOC with 1,405 samples, 1 sample over 600 LOC, all 32 C keywords present vs 27 in Juliet). - number: Over 90% initial GPT-3.5-turbo compilation rate; cost 0.002 USD per 1K tokens (~200 USD total, generated in 24 hours). - number: 109,757 samples (98%) compile with `gcc -lm` alone; 2,243 (2%) require 10 external libraries. - number: 106,139 samples verified by ESBMC; 5,861 excluded (786 failed clang compilation, 5,075 crashed ESBMC). - limits: Functional correctness is not assessed. Verification is bounded to one loop unwinding and 30 seconds; successful/unknown programs may contain undiscovered bugs. Hardware affects timeouts. 5,861 generated programs were not classified due to Clang compilation failures or ESBMC crashes. CWE-242 reports were omitted because reported locations could point to a library declaration. Code and results files were not inspected. - how much was read: most of it read - source - record id: `the-formai-dataset-generative-2023` - TLA+ ZooKeeper reliability (2023) - title: Leveraging TLA+ Specifications to Improve the Reliability of the ZooKeeper Coordination Service - what: Builds three linked TLA+ specifications for ZooKeeper/Zab: an abstract protocol specification, a detailed system specification serving as a super-document, and a test specification that drives explorative implementation testing. The protocol model captures Zab ordering and recovery; the system model refines implementation details and uses interaction-preserving abstraction to make TLC checking feasible; the test model and Met conformance framework generate traces for the Java implementation. The authors reproduce deep ZooKeeper bugs ZK-2845 and ZK-3911 involving accepted-log inconsistency or committed-log loss, and merged the specifications into Apache ZooKeeper. - number: protocol configurations reach tens of thousands of states in seconds - number: system configurations reach billions of states and some exceed 24 hours under stated limits - number: two deep bugs reproduced: ZK-2845 and ZK-3911 - number: trace length limit 100 in simulation; model-checking time limit 24 hours and disk limit 100 GB - limits: The objective is usable specifications and automated testing rather than full verification. TLC explores bounded configurations, trace lengths and resource limits; the system-level state space can become intractable. Conformance depends on the abstraction and instrumentation mapping between model actions and implementation events, and does not prove all Java code behavior. - how much was read: most of it read - source; notes: ## ZooKeeper TLA+ specifications (2023) - record id: `tla-zookeeper-reliability-2023` - Cosmos DB TLA+ (2022) - title: Understanding Inconsistency in Azure Cosmos DB with TLA+ - what: # Whole-paper read: Understanding Inconsistency in Azure Cosmos DB with TLA+ (authored by agents unless marked 🧑) A. Finn Hackett, Joshua Rowe and Markus Alexander Kuppe; arXiv2210.13661v1,24October2022,11pages. This is a client-facing behavioral model and industrial explanation case study. It reports390lines of core specification developed over2person-months,24non-type properties checked by bounded exploration/simulation, two documentation issues and an abstract explanation of an Azure outage. The service implementation is represented through expert feedback, without automated conformance checking. [Q0,Q1,Q11,Q19] ## Model and assumptions The four conceptual variables are an ordered write log, a universally replicated prefix readIndex, a durable majority-replicated prefix commitIndex, and a failover epoch. Their relationship is readIndex≤commitIndex≤Len(log). Replication nondeterministically increases either prefix; data loss truncates only the non-durable suffix and increases epoch. The log abstracts replica/region arrangements rather than enumerating them. The retrieved source additionally carries the configured write level as an unchanged state variable; the paper calls it a constant. [Q3,Q4,Q5,S1] One write consistency level is configured for a deployment; individual reads may select an allowed weaker level. The model concerns individual single-key operations. Transactions and optimistic concurrency are outside its scope. Bounded staleness models operation counts, omitting the wall-clock bound. VersionBound limits unreplicated entries for all write levels; StalenessBound separately limits non-durable entries when bounded-staleness writes are configured. These abstractions should not be mistaken for a proof covering arbitrary implementation arrangements merely because the model does not explicitly contain servers. [Q3,Q5,Q14] A write begins by appending its key/value and obtaining an epoch/checkpoint token. It can later succeed or fail. Failure need not delete the value: the request can have taken effect even if its response was lost. Weak reads can see the value while the write remains pending. Successful strong writes require durability. An epoch change invalidates all old session tokens in this model; authors acknowledge that real data loss need not invalidate every token. [Q6,Q7,Q9,Q18] GeneralRead(key,index,allowDirty) returns the latest mapping at or before index, or a not-found marker if absent. If dirty reads are allowed, it also permits later matching entries. Strong reads use commitIndex without dirty values; bounded-staleness reads use commitIndex with dirty values; session reads use max(token.checkpoint,readIndex) with dirty values and require a current-epoch or null token. The client updates its token with the maximum prior/read checkpoint. Prefix/eventual reads both use readIndex with dirty values. This last equivalence concerns single reads/writes, not transactions. [Q8,Q9,Q10,Q14] ## Validation and its limits Authors report10liveness and14safety properties across4distinct levels, excluding basic typing, using exhaustive logs up to length6 and depth-first random traces up to100steps. Examples include durable committed prefixes, read-your-writes, monotonic reads, staleness bounds, token lifetime and eventual write completion. The paper gives no complete per-property run logs, state counts, runtimes or mapping of every property to each exploration bound. These are bounded model checks and sampled traces, not a parameterized proof or implementation test. [Q11] The pinned artifact gives useful additional scope. MCCosmosDBProps uses two keys, three values, StalenessBound2,VersionBound3,Len(log)<3 and epoch<3. Its configuration labels several temporal checks “very very slow (1hr+)” and three “overloads TLC (out of memory), probably holds”. SimCosmosDBProps has a finite natural-number replacement0..100 and omits some costly properties. These comments are evidence about the retrieved snapshot, not proof that the paper's historical runs failed; neither do they substantiate completion of every stated check. The history specification assumes weak fairness of the disjunction WriteSuccess∨WriteFail; the simulation specification omits that fairness clause. Thus completion claims depend on the specification and bounds actually used. I did not establish an unbounded per-write progress theorem. [S2,S3,S4] CosmosDBLinearizability maps strong read results to an atomic dictionary view. A commitIndex jump can correspond to multiple dictionary writes in one abstract step. The saved configuration asks for DictSpec under epoch<3 and Len(log)<4. This supports the paper's model-level refinement argument; I did not independently establish a full invocation/response-history linearizability theorem. The old-model refinement goes from the older model to this newer behavioral specification, so old behaviors are included in the new model. The claimed strictness additionally rests on the newer model admitting data loss and dirty reads omitted before. The retrieved mapping fixes two regions/two clients per region/one write region and constrains history length<5. I inspected the mapping, not its old-model dependency closure. [Q12,S5,S6] The read hierarchy is checked while holding write consistency fixed and considering all session-token choices. Comparing differently configured write levels does not give the same inclusion: strong writes can admit more pending non-durable values than bounded-staleness writes. The four retrieved adjacent read-refinement modules use fixed write configuration and small finite state constraints; no unbounded theorem proof is attached to their THEOREM declarations. [Q13,S7] ## Concrete cases and industrial results The first documentation issue is a claim that prefix reads cannot observe A,C or B,A,C after sequential writes A,B,C. A,C is possible simply by interleaving reads around writes. For B,A, use five replicas: A reaches1,2,3;B reaches1,4,5;one read hits5 and another2 before replication catches up. Each write reached a local majority, yet the client seesBthenA. These are allowed service behaviors, not newly demonstrated implementation failures. Authors report the documentation was rewritten. [Q14] The second issue concerns regional guarantees. Bounded-staleness operations in one region may appear strongly consistent until a region failover loses or exposes older data; they lack global durability. Authors report their argument was confirmed for individual operations, while regional documentation bullet points remained because transactions distinguish prefix and eventual behavior. The paper's section title about regions must be read with that explicit restriction. [Q14] The outage affected thousands of subscriptions over28days and was noticed after26days. FrontDoor wrote task data, then asked Worker to read it. Session consistency was configured, but the processes used separate/unconfigured session tokens. Their cross-process interaction therefore had only eventual-read guarantees. A routing change exposed the existing semantic problem through latency. A PlusCal model reproduces a missing read before replication; forwarding FrontDoor's token to Worker removes the modeled error according to the authors. The original postmortem author confirmed the explanation made sense. This is an abstract causal explanation and proposed/rechecked model fix, not evidence that the fix was deployed or that every production trace was matched. [Q15,Q16,Q17] Other cases show why the model has two-step writes: a weaker read can observe an incomplete strong write; a failed write can remain durable; and a bounded-staleness read under strong-write configuration lacks the separate bounded-staleness write throttle. Although the paper says strong consistency has no bound on in-progress requests, the model still has VersionBound on partially replicated entries; completed replication permits more pending requests. These are distinct quantities. Expert review caught an earlier incorrect atomic-write abstraction, which was rewritten and rechecked in2–3days. [Q18,Q19] ## Printed text versus pinned source The displayed WriteCanSucceed on paperp4 checks token validity and, for strong writes, token.epoch=epoch. It omits the checkpoint≤commitIndex condition that the immediately following prose says is required. The pinned CosmosDB.tla includes that durability conjunct. Thus the displayed formula and prose disagree; this does not establish that the checked source permits non-durable successful strong writes. [Q7,S1] The outage text's expected formula is eventually workerValue="workerValue", but Listing1writes"taskValue". The source's temporal property uses eventually workerValue="taskValue" and its terminal-state property checks the same literal. I treat the printed literal as a presentation mismatch and preserve the intended missing-task-value scenario rather than presenting the printed formula as a valid test of that scenario. [Q16,S8] These comparisons use a retrieved, pinned repository snapshot, not an asserted reconstruction of the exact2022experiment checkout. No author code or proof/checker was run. [S9] ## Contribution and reading scope For this database, the contribution is a reusable specification of a distributed dependency's visible behavior and a concrete demonstration that model correctness also depends on choosing faithful abstractions. Its relevance to LLM-written specifications is my inference: evaluation should test behavior the model must permit as well as properties it must forbid, record expert corrections, and distinguish bounded self-consistency from service conformance. The paper has no Verus, Rust or LLM experiment. [Q19,Q20] Read all11pages,28reference entries, every displayed definition, Listing1,all4figures and TableI; inspected all11page renders across the preceding and current context. Read the complete core/property/read-wrapper/client/linearizability modules, both PlusCal outage modules, both model-check/simulation wrappers and configurations, four read-level refinement modules and their common-pattern constraints, the general-model mapping/configuration, and two repository READMEs.26static files acquired with hashes and Git blob identities. No old general-model dependency closure or broader implementation inspected. Official and inventory PDFs are byte-identical. No author execution/import, solver, compiler, checker or serialized scientific-results loading. - number: 2 person-month development - number: 10 liveness and 14 safety properties across 4 consistency levels - limits: single-keyindividualoperations;countnotwallclock;alltokensinvalidatedonepoch;boundedchecks;expertconformanceonly - how much was read: most of it read - source; notes: P24. Understanding Inconsistency in Azure Cosmos DB with TLA+ - record id: `cosmos-db-tla-2022` - Demystifying and Checking Silent Semantic Violations in Large Distributed Systems (2022) - what: Empirical study of 109 production silent semantic failures from nine systems, followed by Oathkeeper: a runtime checker that learns likely event-relation rules from patched regression traces, keeps rules failing on corresponding buggy traces, filters against the remaining patched tests, and monitors future execution. It detects six of seven selected newer known failures and reports activation-conditioned false-positive rates. This is trace-based inference and runtime detection, not an inductive proof of distributed-system correctness. Records the ticket-sampling population and bias, thirteen study findings, exact before/after regression workflow, event and state instrumentation boundaries, counter-based implication semantics, inactive-rule distinction, finite-window/recheck tradeoffs, all seven newer contracts, cross-case transfer limits, all throughput values and activation-conditioned false-positive denominators. - limits: §2: Semantics are guarantees about output, state and subsequent operations in response to requests, commands, messages, environment changes or time. Silent violations leave the system operational without explicit crash/timeout/exception signals. A later error in another operation can expose the earlier violation. The scope includes system-specific contracts beyond protocol consistency. §3: Tickets must be resolved server bugs above minor priority. Iterative random sampling balances candidate counts across systems; nonproduction/internal-test cases and explicit failures excluded. In-depth 109 cases prefer information-rich discussions among 268 candidates from 747 valid cases in 1,922 sampled tickets out of 38,786 total. Multiple inspectors reconcile classifications. Selection, reporting, survival and Java/C++ biases prevent interpreting percentages as universal production failure probabilities. §4: Reported mean 39% (range 20–57%) is the paper’s average of candidate/valid ratios across sampled systems. Priority labels normalized to blocker/critical/major; 45% of studied cases are blocker or critical. This is historical issue prioritization, not independent measurement of severity. Consequences in Figure 3 differ from the violated guarantee itself and can arise later. §5: 87% of violated semantics explicitly documented (53% API, 24% internal behavior, 10% configuration); remaining 13% implicit expectations. Documentation can be vague/incomplete and is not mechanically enforced. Old semantics means present in first major stable release. Their 68% share has no denominator of all old/new contracts, which authors explicitly did not count; it does not estimate per-contract risk. §5–7: 74% of violations judged locally, sometimes using replicated local session/node state; 26% need global information. This is case classification, not a proof that every locally observed trace represents a consistent distributed snapshot. Safety/liveness classification follows official documentation: 86% safety, 14% liveness. Finite absence of an eventual response alone cannot establish general liveness failure. §6–8: Only 12% assigned familiar memory/race/overflow bug patterns; most are varied system logic/design defects. Two-thirds violate guarantees extending beyond operation return, so single-entry assertions can miss later failures. Existing semantic test coverage in 73% of cases does not cover all triggering combinations. MongoDB’s production assertions correlate with its lower silent-failure fraction, without causal identification. §8.3: Observability percentages derive from only 34 tickets explicitly describing how users noticed the problem, not all 109. Every observed case had already been discovered, creating survival bias. Overlapping APIs and state-query APIs are recommendations from these observations rather than evaluated interventions. §9.1–9.2: Users provide build/test configuration and past failure commit IDs/test names, optionally classes and compatibility patches. Tool executes patched and preceding buggy versions; regression tests copied to buggy code may require interface fixes. Evaluation instead adds switches to enable/disable old patches. Useful input therefore depends on reproducible old failures and adequate regression tests. §9.2: Java load-time bytecode hooks at method entry, return and exception; state hooks target primitive/collection members in patch-affected classes plus optional user classes. These are selected observable events, not every semantic state or an automatically proved mapping from documentation. Event records contain context such as variable name/location/time. Operation-event uniqueness in the illustrated template uses method signature; paper does not prove per-client/object obligation matching or cross-node ordering soundness. §9.3: Eighteen manually designed templates constrain candidate search to event types and relation structures; seven examples appear in Table 3, full list deferred to technical report. Parameterized matches are likely invariants of the observed traces, not exact/full high-level semantics. No induction, model checker or universal proof validates unseen executions. Templates require domain knowledge but avoid writing each concrete rule manually. §9.3/Algorithm 2: p⇒q uses one counter per distinct ordered event pair. Each p increments outstanding count; subsequent q decrements positive count. Earlier q does not satisfy later p; one q discharges one outstanding p. Zero final counter yields candidate rule. This is stronger than existence of one later q for all p and still depends on event identity/context. Enumeration is affordable for reported test traces, not claimed scalable to arbitrary production histories. §9.4: PASS requires activated antecedent and no contradiction, INACTIVE means antecedent absent, FAIL means contradiction. Retain rules passing patched regression and failing buggy trace; then discard those contradicted by any patched test, excluding start/shutdown/thread-run regions via test markers. Inactivity is not positive evidence of universal correctness; test filtering can retain coincidences and eliminate desirable properties if tests differ in intended scope. Listing 1/Algorithms 1–2, p.100: Printed Verify interface says true means continue scanning; generic Algorithm 1 breaks when scan returns true, while Algorithm 2 returns true for every event. Generic Verify also omits the prescan initialization called for by the interface/example. Record these as apparent pseudocode inconsistencies; no implementation inspected, so do not infer a tool defect. §9.5–9.6: Production checker runs periodically, logs counterexamples, rechecks shortly to tolerate transient violations, and alerts rather than crashes the service. Recent-event window exceeds checking period; time-related events retained by parameter expiry. No formal proof of event completeness, buffer-loss freedom, ordering or preservation of arbitrarily long pending obligations under window expiry. Periodic finite-trace checks and short recheck are practical monitors, not general liveness verification. §9.6–9.8: Survivor optimization stops validating a rule once disproved; related tests prioritized. Ring buffers decouple emission/checking, reuse expired objects and preallocate per event type. Assumptions: contracts expressible through simple event relations, enough good tests, and old patches without major redesign/interface changes. Java/JDK 8, Javassist and JUnit implementation; no Rust/Verus/TLA+ or LLM specification/proof experiment. §10.1–10.2: Eight ZooKeeper, ten HDFS and eight Kafka old failures selected with regression tests. Seven later cases are known bugs chosen for related semantics but different root causes, nine to 34 months later. Six detections show transfer within that selected scope; not prospective discovery of six unknown bugs or recall over all failures. Unseen semantics expected to be harder. ZK-1667 fits a template but training test quality yields wrong rules. §10.2: Mean 0.91 s measured from violation onset after deliberately triggering reproduction soon, not client-operation start or production incident time. Comparator uses Daikon on instrumented state variables, the invariant engine underlying Dinv, and detects one case with noisy rules; not a comprehensive unchanged Dinv deployment comparison. §10.2 additional experiment: 22 ZooKeeper failures sorted by ticket time. Six individual failure-derived rule sets transfer to a median five failures; 16 detect only their own failure within the pool. Figure 11 aggregates a sliding subset as caption states. This assesses overlap in a historical pool, not independent prospective validation or a universal specification. §10.3–10.5: Runtime overhead measured on three stated workloads, not arbitrary production load; checker asynchronous. Only 11%, 66%, 48% of rules activate on ZooKeeper/HDFS/Kafka, respectively. False-positive rates 4%, 9%, 12% apply among activated rules, not all rules, requests or failures. Paper does not report observation duration, confidence intervals or exhaustive false-negative rates. Suggested profiling/banning false rules not evaluated here. Scope: All evidence below is read from the proceedings paper; no tool, benchmark, regression tests or repository executed. Artifact availability/functionality/reproduction badges are printed on p.91; they are not an independent reproduction by this reader. - how much was read: most of it read - source - record id: `demystifying-and-checking-silent-2022` - DuoAI (2022) - title: DuoAI: Fast, Automated Inference of Inductive Invariants for Verifying Distributed Protocols - what: Infers inductive safety invariants for Ivy distributed-protocol models by filtering formulas on reachable simulation states and refining candidates with Ivy/Z3. A finite directed implication graph organizes formula weakening. Runs top-down weakening, bottom-up universal-core strengthening and a universal-only search in parallel. Reports26of27protocols solved, including five Paxos variants; vertical Paxos exhausts memory. Completeness is relative to an adequate finite decidable formula space and graph/optimization correctness, with a reader counterexample to the printed arbitrary-alternation decomposition extension. Supplies exact/approximate implication graphs, sample-preserving coverage theorem, monotonic-refinement proof, counterexample-pruned powerset search proof, quantified-variable/type ordering, mutual-implication optimizations and all benchmark outcomes. Explains which refinement wins per protocol, why hundreds of existential candidates can stall SMT, what human choices baselines require, and how specification/retrofitting differences affect claims of automation. Reader checks distinguish sufficient invariant inference from implemented-system verification and identify a quantifier-dependency flaw in a printed optimization lemma. - number: solves 26 of 27 protocols including Paxos variants; Vertical Paxos runs out of memory - number: Fast Paxos takes 26,979 seconds (about 7.5 hours) - number: solves all but 1 of 27 protocols (Vertical Paxos unsolved) - number: evaluation permits a week per tool and protocol - limits: Inputs are an Ivy protocol transition model and safety property, not arbitrary executable distributed code. Trusted Ivy/Z3 checks initialization and transition preservation of the conjunction and safety implication for all modeled instances, rather than bounded simulation being the certificate. Scope is safety; no liveness, crash-recovery implementation refinement, verified Rust/Verus proof or LLM-agent evaluation. Sampled states are assumed genuinely reachable; incorrect simulation could discard needed true invariants and compromise completeness, though final sound SMT checks still guard returned results. Finite formula search space uses prenex DNF, relational predicates/equality; extension to uninterpreted functions referred to DistAI. Bounds are quantified variables per sort, max_exists, max_literal, max_and and max_or. Tautologies/contradictions excluded. Nonempty sorts are implicit in universal-to-existential implication. Quantifier alternation must follow a fixed sort order, inferred partly from axioms or supplied by user; absent user choice multiple orders are tried in parallel. The actual full verification condition, including protocol axioms/transitions, must remain in the intended decidable fragment. Exact graph is transitive reduction of strict implication over finite formulas; syntactic implementation instead uses five sound weakening shapes and may connect equivalent formulas. Lemmas1–5 establish claimed acyclicity, implication and coverage of nontrivial formulas from selected roots. No theorem says every semantic implication is an edge/path in the approximate graph; coverage plus sound directed weakening is what enumeration/refinement need. Enumeration checks original variable-length states at multiple finite instance sizes, rather than DistAI incomplete fixed-length subsamples. A candidate is retained if it holds on all sampled states; descendants can be skipped if dominated by an existing candidate. Theorem1 says every true invariant in S is either itself retained or has a stronger retained ancestor. Retained candidates can still be false on unseen reachable states; simulation removes candidates, not proof obligations. Top-down refinement checks conjunction plus safety, replaces a failed candidate by noninvalidated immediate successors not already dominated by a reachable ancestor, and returns NotProvable if safety fails. Reachable ancestor here means a graph path avoiding invalidated nodes, not a reachable protocol state. Theorem2 maintains domination of every member of an adequate inductive solution; finite formulas each invalidated at most once give finite refinement assuming every solver check terminates. Bottom-up first weakens all universal candidates until inductive, explicitly ignoring safety, to obtain universal core. noncore includes existential candidates AND universal candidates not in the core whose inductiveness may require existential support. Enumerates subsets by size, calls top-down on core plus subset, and skips subsets holding on an already returned negative prestate. A failed safety-induction counterexample is a possibly unreachable state that invariants must exclude; strengthening can remove it. Theorem3 eventually tries all noncore, reducing to full top-down, conditional on correct pruning and finite solver decisions. Relative completeness is not a guarantee to prove every correct distributed protocol. Needed inductive solution must be expressible in chosen finite S and decidable fragment, samples/graph/optimizations must preserve it, solver calls must terminate, and resources must suffice. NotProvable means this search space cannot prove the property, not the protocol unsafe. Increasing size in round-robin has no practical resource bound; vertical Paxos memout is concrete failure despite expert invariants existing. Defaults max_literal4,max_or3,max_and3,max_exists1 unless safety already has>=2existentials; quantified variables per sort initially maximum arity occurrence in any relation. On failure increments literal/OR/AND/existential/per-sort-variable bounds round-robin. Runs a third max_exists0top-down worker along with general top-down/bottom-up; benchmark walltimes benefit from parallel simulation/refinement and possible sort-order searches, with no separate CPU-time/worker-count-normalized comparison. Mutual implication optimization suppresses Q if retained P1…Pk jointly imply Q, then reintroduces Q if a supporting candidate invalidated. Equivalence decomposition suppresses a formula permanently only when conjunction of smaller formulas is truly equivalent. Examples separate two independent existential witnesses, same witness, and distinct witnesses: existential dependency matters to logical validity, not just speed. Decomposition can require representation of each component within allowed search space. Reader counterexample to Lemma6 claimed arbitrary-alternation/skolemization extension: sorts A,X each{0,1}, fixed order A before X. Let p(x) iff x=0, q(x) iff x=1, r(a,x) iff a=x. F=exists a forall x.((p(x) AND q(x)) OR r(a,x)) is false. The selected conjunction p(x) AND q(x) has two components sharing no existential variable. Claimed components exists a forall x.(p(x) OR r(a,x)) and exists a forall x.(q(x) OR r(a,x)) are both true, choosing a=1 and a=0 respectively. A globally chosen existential witness in the other disjunct cannot be chosen separately. Proof for forall-X exists-YZ is valid under its independence conditions; extending it to arbitrary alternation is unsupported and false as stated. Restrict prefix/dependency structure or preserve shared Skolem choices. This challenges printed optimization/completeness scope, not observed artifact behavior; final independent sound SMT checking can still reject unsafe outputs. Printed Lemma6 proof also calls f/g disjunctions of component literals although components of original conjunction must remain conjunctions for Eq19. This is a terminology/algebra transcription issue for multi-literal components, separate from the arbitrary-alternation counterexample. Lemma5 existential-root argument omits stated no-OR root condition; treat as proof detail needing completion rather than infer a specific artifact failure. Printed Algorithm2 loops only while candidates nonempty, then returns NotProvable; it omits explicit check for already-inductive safety when no strengthening candidates exist. For an unchanged Boolean b initiallyfalse and safety notb, an empty candidate set suffices but displayed procedure returns NotProvable without checking. Algorithm3 likewise needs defined empty universal core when Procedure1 returns no candidates. These are boundary/pseudocode qualifications; do not treat them as reproduced implementation failures. This empty-input example concerns the displayed procedure: default enumeration may retain not-b, and Theorem1 prevents an empty candidate set when a nonempty sufficient II-star is represented. It is not an end-to-end failure of default search. The O(m^2) discussion counts a possible pairwise invariant-dependency workload, not a formal polynomial bound on quantified SMT runtime. Solver cost can dominate even with few formulas. Paper compares14needed Paxos invariants verified in<1s, +10correctunnecessaryalternating invariants5min,+20over3h;149universal Chordinvariants8s. This motivates existential sparsity but is not repeated statistical scaling evidence. Evaluation27small Ivy models (21–123LOC,2–13relations),15universal-sufficient and12requiring existential solution in their presentations. Tools SWISS,IC3PO,FOL-IC3,DistAI,UPDR,I4, week cutoff. Fail,toolerror,Z3error,memout,timeout distinguished; no common success criterion should collapse these. Walltimes single reported numbers, no repeated-run distributions/seed sensitivity/memory measurements or solver-query-count ablations reported. Hardware paper reports DellPrecision5829,4.3GHz28-coreXeonW2175,62GBRAM,512GBSSDPro600p,Ubuntu18.04; reported core description preserved without independent hardware validation. Baselines configured with their documented settings: SWISS uses own published formula bounds per protocol; I4/IC3PO need finite-instance sizes, I4 starts minimum and increases on failure; FOL-IC3 runs universal/default modes and reports first success. Tablefootnotes attribute FOL-IC3Chord difference to threebugs in a competitor file and IC3PO successes to retrofitted protocols/manually supplied alternating clauses. These are DuoAI authors claims, not independently audited competitor defects or controlled equal-input performance results. Concurrent PFOLIC3 excluded from main comparison; its failure rates cited from other work, not own experiments. DuoAI solves26/27,15/15universal and11/12existential, fastest outright on15; SWISS19,IC3PO18,FOLIC318,DistAI14,UPDR14,I412. No competitor solves verticalPaxos here. Not every row is a speedup: Chord200.9s vsIC3PO17.1/DistAI58; locks/tablecontain subsecond differences and learning-ring alternatives. Paxos60.4s vsSWISS16665s(~275.91x),flexible78.7vs28337(~360.06x); comparison is measured selected tool/model configuration, not universal algorithmic superiority. Universal-only worker wins all15universal cases; top-down winsfive simpler existential cases and bottom-up wins hybridreliablebroadcast plusfivePaxosvariants. MultiPaxos615candidates/581alternating butonly2needed noncore; Paxos/flexible/multi/stoppable require subsetsize2,fastsize6. Vertical expert invariant8literals reduced7,60predicates/120signed gives search order120^7≈3.58e14; fast38predicates/5literals gives76^5≈2.54e9. These coarse powers are magnitude estimates ignoring syntax/equivalence, not measured exhaustive candidate counts. Implementation6.1KC++LOC(gcc7.5.0),2.3KPythonLOC(3.8.10) simulator. No machine-checked metatheory or independently executed artifact. Proof soundness/trust distinct from search completeness and practical automation; returned inductive safety certificates do not connect protocol abstraction to concurrent network code. Table1 has four IC3PO successes, three failures and five memory failures among the12existential rows. The §8,p.496 prose instead says six memory failures; the table, not that narrative count, is preserved in structured results. This discrepancy does not change IC3PO total18successes. - how much was read: most of it read - source; notes: papers inspected - record id: `duoai-2022` - Endive (2022) - title: Plain and Simple Inductive Invariant Inference for Distributed Protocols in TLA+ - what: Samples candidate lemmas from a predicate grammar the user gives. Checks them on small finite instances. Greedily picks the lemmas that remove counterexamples to the induction step. A separate TLAPS proof is still needed for the protocol at unbounded size. - number: Invariant for 25 of 29 protocol benchmarks - number: Table I has 30 rows incl. MLDR; 2 rows have no TLA+ translation - number: Paper's Endive rerun in IC3Syn solves 24 shared cases - number: endive solved 29 benchmark protocols plus MLDR; MLDR row: 2,025 s, 7,751 CTIs, 1,272 s checking, 651 eliminated, 102 s CTI generation (Table II) - limits: The generated invariant is not guaranteed to be truly inductive, because counterexamples are sampled at random. Needs a grammar from the user. - how much was read: most of it read - source; notes: Endive: TLA+ candidates, followed by an unbounded proof; notes: papers inspected - record id: `endive-2022` - FRET realizability (2022) - title: Realizability Checking of Requirements in FRET - what: It checks whether requirements can be implemented, using Lustre and JKind/Kind2. It finds minimal unimplementable sets of requirements with delta debugging, and shows counterexample traces of the environment. - number: aircraft study: 11 requirements; removing the hover-initialization requirement made a conflicting requirement realizable - number: infusion pump: 12 requirements rewritten to 26 scoped ones, 8 minimal unrealizable cores - limits: This is a technical report, not the CAV 2022 version. Results depend on which variables the environment controls. - how much was read: most of it read - source - record id: `fret-realizability-2022` - Invariant inference with quantifier alternations (2022) - title: Inferring Invariants with Quantifier Alternations: Taming the Search Space Explosion - what: Presents a PDR/IC3 invariant-inference algorithm that learns first-order invariants with quantifier alternations. It lifts quantifier-prefix choice to the inductive-generalization level for parallel breadth-first search, introduces k-term pseudo-DNF (k-pDNF) matrices, and combines EPR restrictions, incremental SMT, multi-block generalization and heuristic may-proof-obligations. The IG procedure partitions prefix space into categories/workers, orders prefixes by quantifier depth/alternations/universals/existentials, shares constraints only from immediate sub-prefixes, and interleaves separation with SMT checks of Init and relative inductiveness. k-pDNF represents a negated cube followed by up to k−1 cubes, expressing implication-shaped Paxos lemmas more compactly than CNF/DNF. The complete P-Fol-Ic3 algorithm has learning and heuristic tasks, optional EPR edge restrictions, and solver robustness via restarts, parallel Z3/CVC5, and incremental assertions. Evaluation covers 30 safe distributed-protocol problems, including Paxos variants, with ablations for pDNF, EPR, incremental SMT and multi-block generalization. - number: 30 benchmark problems - number: 5 trials per algorithm; 16 threads, 20 GiB RAM, 6-hour timeout - number: paxos-epr 920 s (5/5), flexible-paxos-epr 418 s (5/5), multi-paxos-epr 4272 s (4/5) - number: k=1 universal and k=3 non-universal pDNF settings - limits: Benchmarks are safe transition systems with known human invariants; inference success is not a refutation method for unsafe systems. EPR requires a user-provided acyclic set of allowed sort/function edges, and prenex separation may need a special push-down mode. Full FOL search can be intractable; the method failed vertical-paxos-epr and several hardest cases, and stochastic parallelism yields variable success. Experiments use fixed 6-hour timeouts and 16 threads/20 GiB; performance and learned lemmas depend on solver behavior, random seeds and prefix restrictions. - how much was read: most of it read - source; notes: P-FOL-IC3: full 30-model experiment recovered - record id: `invariant-inference-with-quantifier-alternations-2022` - Learning to Find Proofs and Theorems by Learning to Refine Search Strategies: The Case of Loop Invariant Synthesis (2022) - what: Proposes self-training AlphaZero-style teacher and solver agents that refine generic nondeterministic programs for automated theorem proving without human proof examples. Demonstrates Looprl on loop-invariant synthesis: abductive strategy code defines a search space, a teacher generates constrained diverse Code2Inv problems, and a solver learns greedy action choices. On 132 single-loop C programs, trained policy solves 61.5% without backtracking versus 39.7% using an untrained-teacher dataset and 18.4% random actions; MCTS with the generic strategy solves all benchmarks but is not the learned no-search metric. Defines choose/event nondeterministic strategy operators, conditional generative teachers, abductive reasoning and bounded event rewards. A strategy induces an MDP whose choice points are actions; AlphaZero/MCTS policy and value heads refine both teacher and solver. Looprl compiles an OCaml DSL to RL environments, provides graph-format UI/debugging, integer-linear-arithmetic abduction, and parallel Gumbel AlphaZero. Teacher constraints include invariant shape, conditionals, usefulness of guard and equalities; solver reward penalizes proof steps. A complete run uses 20 teacher +20 solver iterations, 1.6M-parameter graph transformers, 50K solver training problems, and one16-hour RTX3080 run. Supplement findings: Appendix A expands the solver strategy to disjunctions of at most three atomic candidates, optional candidate strengthening, lazy constant metavariables with global constraints, and conjectures from preserved linear combinations, relaxed guards and unmodified initial variables. Conjecturing costs -0.3, abduction -0.2, each counted at most four times, with successful reward at least zero. Figure 7 provides all 94 lines of the strategy and a pending stack encoding. This makes the hand-designed search space and shaped reward explicit; the neural policy chooses within it. Appendix B/Table 3 lists 15 sampled constraint types; penalties are -0.5 except the last four at -0.2, and successful reward is at least -0.5. Table 4 hard constraints require correctness, nontrivial/nonredundant formulas, satisfiable invariants, entering the loop, and a restricted loop-terminates test. The paper explicitly says that test rules out only a subset of nontermination cases. It must not be described as a general termination guarantee. Constraint combinations and correlations are partly hardcoded; formula/program templates and parameter-abduction shape the generated distribution. Table 5 gives 12 final transformations, including irrelevant guards/formulas, commuting or shuffling instructions, comparison normalization, moving assumptions and weakening the final assertion. An application is cancelled if it violates a hard constraint or increases soft violations. Table 6 gives Code2Inv problems 3,7,93,110 and explanatory invariants: disjunctive x<5 or y≤z; x-y≤10; 3i=x+y with i≤n; i-s=1 with i≤n+1 or s=0. Table 7 prints three generated programs/invariants, with final transformations disabled for clarity and invariants hidden before solver training. They are illustrations, not additional held-out measurements. Appendix F uses Fourier–Motzkin elimination on negated CNF clauses in linear integer arithmetic. For a single clause, negations of derived consequences suggest sufficient assumptions; contradiction can prove validity. For multiple conjuncts, the implementation returns the union of clause-local suggestions rather than all conjunction combinations. F.2 explicitly withdraws the guarantee that any suggestion implies the entire input formula. Subsequent proof checks and recursive refinement matter; treating every abduction result as a complete proof would overstate the method. Table 8 specifies teacher 20×8000 generation episodes, 800 validation episodes/iteration, max 60 steps, 64 MCTS simulations; solver 20×20000 episodes, 5000 validation/iteration, max 12 steps, 32 simulations. Both use 600 asynchronous/parallel episodes, 128-dimensional embeddings, four attention heads, 6 state/3 action/1 combiner blocks, state/action encoding limits 80/12, 16 identifier IDs and 256-node MCTS cap. Teacher/solver dropout is 0.05/0.1, batch size 400/300, learning-rate maximum 5e-4/3e-4, maximum epochs 6/1. The last five teacher iterations and 10000 extra noiseless generated problems supply solver problems. The 20000 solver episodes per iteration are distinct from the inherited 50000-problem training pool. Appendix H uses separate Dynamic Graph Transformers for state probe and each action, a combiner for softmax action scores, and a value head for outcome/event predictions. AST Polish-order tokens retain syntactic edges, data/control-dependence edges and numerical comparison edges. Identifier IDs are randomized per choice point; overflow names share a flagged ID. H.2 describes binary constant encoding generally, but Table 8 sets const-emb-size=0 for both reported agents, disabling that binary value encoding. Numerical comparison edges remain enabled. Search exceeding state/action/step limits fails or discards actions; no unrestricted completeness claim follows. Figures 8–9 expose policy probabilities, outcome/event estimates and MCTS statistics. The solver example gives prior 0.81 for abduct x>0 and predicted value 0.46 versus described true shaped reward 0.4; teacher screenshot predicts success 0.97 and soft-constraint violation risk 0.31. These are individual UI examples, not calibration experiments. Figure 7 as printed has inconsistent/unbound names in prove_preserved (suggs/assum/action) and proved_invs versus invs_proved; it is explanatory pseudocode, not a standalone executable listing. No author code was inspected to resolve the differences. - limits: The proof domain is Code2Inv:132 C programs with one loop, linear integer arithmetic and selected conditionals/assumptions/nondeterministic tests. The authors leave other domains for future work; general theorem-proving claims are a design vision, not demonstrated coverage. The learned result evaluates greedy highest-policy actions with no search/backtracking. It measures ability to generate an invariant of minimal size; MCTS can solve all benchmark problems, so no-search success is the reported challenge. An invariant is accepted through the benchmark/verifier pipeline; the paper does not establish semantic robustness beyond those checks. Teacher generation uses manually defined constraints and validity-preserving transformations. Constraints can be incompatible; rewards are shaped so success beats failure. Generated problem distribution and benchmark generalization depend on strategy design, not an unrestricted theorem distribution. Training protocol uses two random seeds for error intervals; one complete run takes about16h on specified hardware. No comparison across many independent training runs, hardware settings or random datasets is reported. The approach formalizes expert strategy fragments and learns choices; it does not produce Lean/Coq/Rust/Verus proof terms or TLA+ specifications in this paper. Appendix B/Table 3 lists 15 sampled constraint types; penalties are -0.5 except the last four at -0.2, and successful reward is at least -0.5. Table 4 hard constraints require correctness, nontrivial/nonredundant formulas, satisfiable invariants, entering the loop, and a restricted loop-terminates test. The paper explicitly says that test rules out only a subset of nontermination cases. It must not be described as a general termination guarantee. Constraint combinations and correlations are partly hardcoded; formula/program templates and parameter-abduction shape the generated distribution. Appendix F uses Fourier–Motzkin elimination on negated CNF clauses in linear integer arithmetic. For a single clause, negations of derived consequences suggest sufficient assumptions; contradiction can prove validity. For multiple conjuncts, the implementation returns the union of clause-local suggestions rather than all conjunction combinations. F.2 explicitly withdraws the guarantee that any suggestion implies the entire input formula. Subsequent proof checks and recursive refinement matter; treating every abduction result as a complete proof would overstate the method. Appendix H uses separate Dynamic Graph Transformers for state probe and each action, a combiner for softmax action scores, and a value head for outcome/event predictions. AST Polish-order tokens retain syntactic edges, data/control-dependence edges and numerical comparison edges. Identifier IDs are randomized per choice point; overflow names share a flagged ID. H.2 describes binary constant encoding generally, but Table 8 sets const-emb-size=0 for both reported agents, disabling that binary value encoding. Numerical comparison edges remain enabled. Search exceeding state/action/step limits fails or discards actions; no unrestricted completeness claim follows. - how much was read: most of it read - - record id: `learning-to-find-proofs-2022` - Zeebe Raft property tests (2022) - title: Bulletproofing Zeebe concurrency bugs - what: An engineer at Camunda describes how the team tests the Raft (replication and leader-election protocol) module of the Zeebe workflow engine. Random sequences of events (messages, drops, restarts, clock ticks) are run under a deterministic scheduler, and after each step the code checks two safety properties; for liveness it runs extra failure-free steps and checks that a leader exists and all records are replicated. - number: 2 safety properties and 2 liveness properties tested - number: 6 bugs named: #5360 and #5356 (safety, critical), #8324, #10180, #10202, #10545 (found after liveness tests) - number: @Property(tries=10) per test, run on every CI build, pull request and main-branch commit - limits: Testing gives no guarantee: the tests sample a fixed number of random executions per build. Liveness is checked only when failures stop, and the blog says the system makes progress 'only in specific cases'. Only used on the Raft module and on randomly generated process models. - how much was read: most of it read - source; notes: Bulletproofing Zeebe concurrency bugs (2022) - record id: `zeebe-raft-property-tests-2022` - DistAI repository (2021) - title: DistAI: Data-Driven Automated Invariant Learning for Distributed Protocols - what: DistAI simulates a protocol, learns candidate invariants from traces, and refines them with Ivy and Z3 counterexamples until validated. The README gives commands for two-phase commit, distributed lock, database chain replication and blockchain. - number: solves 13 of 14 protocols; Paxos fails - number: verifies 13 common distributed protocols automatically - number: 14 protocols - number: Suite: 14 distributed protocols written in Ivy (e.g. Paxos, two-phase commit, Chord, Ricart-Agrawala) - limits: The README warns that its parser does not support the full Ivy 1.7 grammar; repository instructions and reported outputs were not rerun. - how much was read: most of it read - source - record id: `distai-repository-2021` - IC3PO symmetry verifier (2021) - title: On Symmetry and Quantification: A New Approach to Verify Distributed Protocols - what: Introduces IC3PO, symmetric incremental induction extending finite-domain IC3/PDR. It infers the quantifier prefix of inductive invariants from protocol symmetries, then finds a finite cutoff whose proof generalizes to all instance sizes. Logical orbits map symmetry classes to quantified clauses; symmetry-aware clause boosting avoids searching fixed quantifier templates. A cutoff procedure checks that orbit representatives cover all relevant instances. IC3PO handles universal/existential alternations and produces compact quantified invariants automatically. - number: Evaluation set 29 protocols; IC3PO solved all 29, versus I4 13, UPDR 14, fol-ic3 23 - number: 10 protocols solved by all: IC3PO total 232s/85 invariant assertions/12,160 SMT queries - number: IC3PO 1-hour timeout, 32GB memory, Intel Xeon X5670; Z3/Yices/CVC4 backends - number: ToyConsensus(3,3,3): IC3PO 192 SMT calls and 13 CTIs versus I4 7,814 calls and 443 CTIs - limits: Focuses on parameterized protocols whose sorts are sets of indistinguishable constants; ordered sorts and integer arithmetic are future work. Safety only, not liveness. Finite cutoff correctness depends on symmetry assumptions; evaluation uses bounded protocol models and SMT solver timeouts. - how much was read: most of it read - source; notes: papers inspected - record id: `ic3po-symmetry-verifier-2021` - Metastable failures (2021) - title: Metastable failures in distributed systems - what: Names a failure pattern in which a trigger (for example a short network outage) pushes a system into a bad state that a feedback loop (for example client retries) keeps going after the trigger is gone. Gives four production examples and a vocabulary: stable, vulnerable, metastable, trigger, sustaining effect, hidden capacity. - number: Retry example: 280 queries per second (QPS) plus a 10 s outage leads to 560 QPS of retries; recovery needs load under 150 QPS or retries under 20 QPS - number: Look-aside cache example: advertised capacity 3,000 QPS, hidden capacity 300 QPS; losing a 90% hit-rate cache gives 10x query amplification - number: Link imbalance example went unexplained for more than two years; fix was a single line in the connection-pool policy - number: A geo-distributed system with added retries and failover had worst-case work amplification over 100x - limits: A vision paper from operating experience: no formal definition of the property, no tool, no measured evaluation; the authors list how to find unknown metastable failures as open. - how much was read: most of it read - source; notes: ## Metastable failures (2021) - record id: `metastable-failures-2021` - Metha: Network Verifiers Need To Be Correct Too! (2021) - what: Metha generates network configurations, runs them on network analyzers and real-router images, and compares converged routing tables to expose model discrepancies. Semantic constraints activate meaningful protocol computations; minimum/middle/maximum parameter reduction and PICT pairwise tests cover selected feature interactions. Iterative delta debugging isolates all inclusion-minimal failure-inducing feature sets under explicit monotonicity/outcome assumptions. Records grammar versus semantic constraints, finite topology/parameter/test domain, exact coverage definition, iterative delta-debugging proof/complexity/masking limits, practical convergence oracle, eight detailed model-mismatch examples, complete bug/ablation/localization counts and commits, baseline parser exception and simulation-only NV boundary. - limits: §§1,3,7: Differential oracle is actual Cisco/Juniper router software, including vendor quirks and bugs. A discrepancy establishes disagreement with the selected implementation/version, not protocol-standard truth or independent proof that the analyzer is at fault. Developers confirmed 59/62 reports. Oracle fidelity, routing-table extraction, grammar/semantic constraints and comparison code remain trusted. §7: Convergence means ten unchanged routing-table snapshots sampled 10s apart (~100s), a heuristic rather than proof that the protocol has stabilized forever. Transients or oscillations slower than sampling, hidden state, timing/hardware differences and output normalization may affect comparisons. Paper gives no exact Cisco/Juniper image versions or host CPU/RAM. Batfish/NV source commits are cited. §§4–5: Search uses fixed physical/logical topology and provided features, finite recursive grammar, predefined strings, numeric minimum/maximum/one interior value. Pairwise coverage is over selected router×feature/value combinations, not all configurations, numeric values, protocol states, failures, traffic or code paths. Supporting prefix/community lists are not independently fuzzed. §5.1: Full t-wise coverage finds every bug only if each is exposed by at most t interactions of selected value dimensions and test behavior meets the premise. Empirical one/two-feature causes do not prove larger or unseen-boundary bugs absent. PICT yields a covering suite; no proof of globally minimum test count provided. §6/Appendix A: Theorem 1 concerns all inclusion-minimal failing subsets, not globally smallest cardinality or every underlying code bug. Iterative extension lifts the unique-minimal-fault assumption, retaining monotone failure and definite consistent outcomes; proof explicitly invokes monotonicity. No proof every real configuration discrepancy satisfies these assumptions. §6.2: Bugs whose triggers strictly contain an already failing subset cannot be identified separately until the smaller bug is fixed. Different minimal triggers can share a source-code cause; the black-box method cannot deduplicate by code or pinpoint erroneous code. Counts represent developer-assessed/configuration-trigger evidence, not automatically proved distinct code defects. §6.2: Uncached worst case is O(|Q|!); caching unique queued feature sets gives stated O(2^|Q|) bound. Router/tool calls and minimization also cost time. Algorithm 1 prints head(queue) without explicit dequeue; narrative describes consuming the queue until empty. This listing omission is not evidence of implementation nontermination. §§7–8: Routing-table comparison catches crashes and silent control-plane model bugs, but does not directly check packet forwarding, data-plane execution, every proof procedure, solver soundness or live production networks. Only NV simulation was tested; authors infer likely verification impact from its shared model. Table 3 marks confirmation of C-BGP’s three reports as unknown (?). Batfish’s 29 and NV’s 30 reports were confirmed. §8.1: Each of four approaches runs 1794 tests with the same four-router Cisco star features. Semantic baseline finds 16 bugs versus random’s 3; boundary values add one, pairwise adds three. Original Metha misses a random parser bug because OSPF area numbers are fixed; adding that dimension finds it. Discussion’s claim that all random bugs are found with fewer tests needs this augmented input space. §8.1/Figure 5: Coverage measures pairs of feature values, with continuous random numeric values bucketed into minimum/maximum/middle. It is not code coverage or proof soundness. Semantic and random approaches have similar coverage but different bug counts, illustrating semantic activation. No repeated random seeds, confidence intervals or sample variance reported. §8.2: Mean 14.1 extra tests per discrepancy, range 7–58, not per bug; 58 localizes five bugs. Complete testing takes several days; single test averages two minutes, PICT generation six minutes. Full cost includes fault localization; no precisely timed total or resource scaling study. §§9–11: Testing supplements unit/system testing. No universal absence-of-bugs proof or Verus/verified Rust/TLA+/LLM-written proof result. Testing configuration synthesizers is future work. Version and confirmation claims refer to source publication time. - how much was read: most of it read - source - record id: `metha-network-verifiers-need-2021` - SWISS (2021) - title: Finding Invariants of Distributed Systems: It’s a Small (Enough) World After All - what: SWISS searches finite spaces of quantified invariants for RML distributed protocol models. Breadth accumulates small relatively inductive clauses; Finisher searches for a final clause that, with safety, is inductive. Failed SMT verification conditions yield reusable finite-model filters, with symmetry and redundancy pruning, model minimization and parallel search. It proves selected models including Paxos/Flexible Paxos and guided Multi-Paxos, and measures successes, partial progress, costs and failed optimizations against I4 and first-order separators. Reconstructs exact Finisher/Breadth obligations, conditional coverage guarantee and its implementation timeout exception, all three counterexample filters and fast bitset/trie evaluations. Saves every benchmark/table number and failed-protocol result, manual search tuning, full-space versus first-hit timing and all appendix ablations. Distinguishes model/spec correctness from code, safety-relative partial clauses from independent proofs, and the small-world hypothesis from general completeness. - number: solves 19 of 27 benchmarks with a six-hour timeout, median of five runs - number: Paxos and Flexible Paxos take about 4.4 and 5.1 hours - number: Multi-Paxos succeeds only with supplied templates - number: Paxos: around 4 hours on an 8-core machine; with guidance 20 minutes - limits: §§2–3: Inputs are a formal RML protocol transition system, desired safety condition and candidate-space configuration. No executable implementation/refinement theorem is supplied. Encoding an intended protocol so checks remain in extended EPR may itself be challenging and is explicitly outside SWISS scope. Safety/transition semantics, solver/encoder/filter implementation and execution environment are trusted; output invariants can be independently validated but this reading did not replay them. §§2.1–2.4: Proofs address all reachable states of the supplied transition model, not just finite test runs; no liveness proof. Atomic transition/failure/network assumptions come from each input protocol. SDL running example uniquely disallows packet duplication; do not generalize that restriction to every benchmark. EPR satisfiability uses ∃*∀* with no general functions; stratified functions and quantifier alternation are admitted in the extended fragment while preserving decidability. §§3.2–3.4: Finisher checks Init⇒S, Init⇒P, TR∧known∧S∧P⇒S′ and TR∧known∧S∧P⇒P′. Breadth checks initialization and inductiveness of new clauses relative to input clauses and S. Therefore intermediate clauses can be inductive only under safety, not independently proved reachable-state properties. Reader qualification: reuse partial output with its assumptions/joint invariant obligation intact; a partial clause count is not a partial correctness theorem about arbitrary code. §3.4 Claim 1: Ideal Solve succeeds only if a finite chain exists with all earlier clauses in B, final clause in F, each earlier prefix inductive relative to S, and final conjunction including S inductive; sound pruning must preserve valid candidates and VC checks must finish. This is relative completeness for the stated chain/space, not decidability of arbitrary invariant discovery. §§4.3,5.1 reader qualification: Actual implementation retries timed-out Z3 queries with CVC4 and may skip an invariant if timeout persists. Thus practical runs with skipped candidates do not retain the ideal exhaustive coverage guarantee. Skipping rejects potentially useful proofs; it does not justify accepting an unproved candidate. Unknown/timeout and finite search failure do not establish protocol unsafety. §4.1: User supplies formula term/depth limits k,d, quantified-variable limits m,q and quantifier type nesting order to retain extended-EPR VCs. Templates can be auto-enumerated within bounds or explicitly supplied. Auto mode is not no-user-configuration; search grows exponentially. Symmetry pruning imposes syntactic term order/first-variable order, not full logical-equivalence solving. §4.2: True(M) requires P on a model; False(M) requires not-P; Transition(M,M′) requires P(M)⇒P(M′). Filters are derived from failed VC obligations and are sound for candidates satisfying the same obligation context. False models need not be reachable; they are states every acceptable strengthening for that VC must reject, rather than arbitrary actual executions proving a bug. §§4.2–4.5: Precompute base-term evaluations on every variable instantiation in finite models and evaluate candidates with bitwise operations. FastImplies is a sound but incomplete syntactic implication check using subsequences/modulo variable renaming in a trie, with limited universal/existential substitution. allInv retains redundant clauses for filtering while indInv avoids redundant output. Model minimization has a 45-second budget, so a retained model need only be partially minimized. §4.5.2: Partition/randomize B/F and run independent per-thread searches, combining only phase results. Threads do not share counterexample filters or fresh invariants during a phase. This gives good filtering parallelism but replicates SMT model construction and does not establish universal linear speedup. §5.1: Defaults B(d,k,q,m)=(1,3,1,5), F=(2,6,1,6) were adjusted per benchmark after results: increase k/q on exhausted failed searches, shrink B on slow breadth. learning-switch-ternary uses q=0; async lock and sharded-kv enlarge k; no-lost-keys needs q=2. Results include manual tuning and guided Multi-Paxos, not a fixed blind default-only experiment. §§5.1–5.2 Table 1: 27 distinct benchmarks plus duplicate guided Multi-Paxos row. SWISS times are medians of five runs with six-hour budget, on eight-core i9-9900K 3.60GHz/125GB/Ubuntu19.10, eight threads unless noted. Z3 4.8.8 then CVC4 1.8-prerelease –finite-model-find after 45s. Baseline runtime entries are published comparisons without uniform per-baseline dispersion in this paper. §5.2.1: Paxos and Flexible Paxos complete automatically within the configured candidate templates; Multi-Paxos requires correct user templates and unguided run times out in Breadth. Fast/Stoppable/Vertical Paxos remain unsolved, despite introductory phrasing that can sound broader. I4/FOL/SWISS each win on some protocols; existential support alone does not dominate larger mutually inductive universal invariants. §5.2.2: Partial progress counts handwritten invariant clauses logically implied by generated clauses, not percent of execution space, correctness or remaining proof effort. Missing one essential clause can prevent the whole safety proof. Reported 79% Vertical Paxos is 14/17, about 82.4%; preserve printed discrepancy rather than treating prose percentage as exact. §§5.3.3–5.3.5: Table 2 timings enumerate entire template spaces even after finding a proof and start with supplied Breadth invariants. The claimed 20-minute/13.4× guided result concerns proof search stopping, not the listed 2036-second full-template enumeration. Table 3 is a single-thread single-template pruning experiment, not end-to-end automatic-template run. §5.3.4: mn/mn−1 are smallest known chain maximum term sizes, drawn from human or synthesized invariants; minimality is not proved. 25/27 have mn≤8 and 22 are smaller; mn≤6/mn−1≤3 is an empirical rule of thumb. Human comprehensibility implies finite intuition is motivation for a hypothesis, not a theorem guaranteeing succinct invariants. Appendices A–B: Formula-synthesis, BMC and eager invariant accumulation were unfavorable in tested cases; this is workload-specific evidence. Figure 5 uses preselected templates and a logarithmic y-axis. Parallelism experiments deliberately finish the whole F space to remove first-hit variance and analyze longest-running threads, so speedups do not directly imply normal first-proof latency. §§5–7: No production-code performance, LLM-agent experiment, verified Rust/Verus implementation or TLA+/TLC integration is evaluated. An unexpectedly easy Fast Paxos proof uncovered a disabled-action typo in the input spec, demonstrating the need to validate intended behavior despite valid invariants. External artifacts/proofs and source-code current state were not executed or audited. - how much was read: most of it read - source; notes: papers inspected - record id: `swiss-2021` - What's Live (2021) - title: What's Live? Understanding Distributed Consensus - what: Extracts published liveness statements for consensus algorithms and formalizes them in TLA+. It compares what each one assumes and guarantees: 3 link assumptions, 7 server assumptions and 6 progress assertions. It also runs finite TLC experiments. - number: of 31 algorithms in Table 1, only 18 (58%) mention liveness and 11 (35%) give a liveness proof - number: TLC on up to 4 proposers and 4 acceptors in a few hours - number: 31 algorithms/variants are catalogued; 18 state liveness assertions, and only 11/31 (35%) provide some liveness proof. - number: TLA+ TLC illustrations cover up to 4 proposers and 4 acceptors for single-value Paxos; multi-value/larger cases become too expensive. - limits: It is an analytical comparison plus finite TLC runs, not new verified code. Appendix A uses a tick bound. Hours and lines of code are not stated. - how much was read: most of it read - source; notes: What's Live? Understanding Distributed Consensus - record id: `what-s-live-2021` - Aragog (2020) - title: Aragog: Scalable Runtime Verification of Shardable Networked Systems - what: Operators write patterns of bad event sequences. Aragog compiles them into state machines and checks them against live or recorded event streams from networked systems. It splits the work by grouping keys. - number: 8 NAT policies at 5-14 spec lines, 3 firewall policies at 4-6 lines - number: four bugs in the NAT gateway, confirmed by operators - number: over 500,000 events/s for one policy, over 30,000 for eight - number: firewall violation detected in about 70 ms median - limits: Aragog has no correctness proof. It needs accurate event ordering and synchronized clocks. Sampling can miss flows. Operators write the specs. - how much was read: most of it read - source; notes: Aragog: Scalable Runtime Verification of Shardable Networked Systems; notes: Aragog: Scalable Runtime Verification of Shardable Networked Systems - record id: `aragog-2020` - tlaplus-monitor (2020) - title: ONOS TLA+ Conformance Monitoring Tools - what: A customized TLC deployed with Helm. It reads JSON traces from Kafka, updates the model state, and checks TLA+ invariants that users write. It has optional sliding windows. - number: one monitor pod per TLA+ specification - limits: There is no paper, and no bug counts, throughput, size or effort. It checks safety invariants on observed traces only. In the documented example, the predicate accepts decreasing versions despite its comment. - how much was read: most of it read - source; notes: ONOS TLA+ Conformance Monitoring Tools - record id: `tlaplus-monitor-2020` - CoqGym (2019) - title: Learning to Prove Theorems via Interacting with Proof Assistants - what: The complete five-page official PMLR supplement is newly read: A.1 project compilation, A.2 proof-tree reconstruction, A.3 synthetic proof extraction and the full Figure A, then every production of grammar B across three pages. The original 11-page CoqGym/ASTactic main-paper record and evaluation are explicitly inherited. The supplement explains collection bias, the discarded multi-goal traces and why synthetic proofs are conditional proofs in augmented contexts; it supplies no additional success-rate experiment or system soundness theorem. Serializes kernel terms/runtime proof states through modified Coq and SerAPI while leaving the checking core untouched; represents proof trees and complete environments. Project-disjoint train/validation/test split has 43,844/13,875/13,137 proofs. Generates 1–4-step synthetic intermediate-goal proofs, but ASTactic training does not use them. Encoder embeds goal and premises with child-sum TreeLSTM; GRU decoder grows Ltac AST depth-first, attends over premises and predicts grammar productions. Argument generation selects available premise identifiers, classifies integers 1–4, or randomly selects quantified goal variables. Teacher forcing and cross-entropy train individual tactics; beam-search tactics feed DFS with duplicate-state backtracking. Combines learned decomposition with built-in automation or hammer at each goal. - number: about 71K proofs from 123 projects - limits: Compilation success and the manually installed dependency subset select the collected projects.123projects/3061files are the collected corpus, not ecosystem coverage. Tree reconstruction uses one disappearing goal per step. Tactics that affect multiple goals/goal selectors are discarded, with reported loss<1%; the denominator is described as data without a detailed sampling/count audit. Synthetic theorem contexts may include generalized descendant obligations H3/H4. This shortens a conditional proof; it does not establish an unconditional shorter proof of the original theorem or independent novel mathematical evidence. The grammar starts at tactic_expr, permits listed atomic tactics, identifiers and small integers1–4, and lists fixed hint databases. It has no arbitrary Ltac sequencing, custom tactics or arbitrary Coq-term synthesis; restriction is deliberate. Inherited main-paper limits still apply: at most10environmental premises plus local context; teacher-forced supervised tactic training; synthetic proofs not used by the reported ASTactic; search/tactic/time budgets; no modern LLM, distributed protocol, Rust/Verus or TLA+ experiment. Only printed grammar/proof examples were read. No repository, Coq implementation, dataset, generated proof corpus or solver/prover execution was inspected or run. - how much was read: most of it read - source; notes: CoqGym: project separation and checker-owned proof state - record id: `coqgym-2019` - I4 (2019) - title: I4: Incremental Inference of Inductive Invariants for Verification of Distributed Protocols - what: I4 generates inductive invariants for distributed protocols by translating finite instances, running Averroes, removing redundant output and checking the candidate against the possibly infinite protocol in Ivy. - number: seven cases: lock server, ring leader election, distributed lock, Chord ring maintenance, learning switch, database-chain consistency, two-phase commit - number: Seven-model suite: chord_ring_maintenance, database_chain_replication, distributed_lock, leader_election_in_ring, learning_switch, lock_server, two_phase_commit - limits: README workflow requires legacy Python/Ivy/Averroes dependencies and does not report reproduced timings here. - how much was read: most of it read - source; notes: I4 - record id: `i4-2019` - Moderately Complex Paxos Made Simple: High-Level Executable Specification of Distributed Algorithms (2019) - what: Rewrites Multi-Paxos pseudocode as a short executable specification in DistAlgo, translates it by hand to TLA+, and proves safety in TLAPS. It also finds liveness defects when messages are lost. - number: DistAlgo core 51 vs 142 pseudocode lines - number: TLA+ models 154, 157, 217 lines; proofs 4,959, 5,005, 7,006 lines - number: TLAPS obligations 4,364, 4,517, 5,580; 590, 569, 781 s on i7-4720HQ - number: 77.822 ms per request with 10 processes - limits: The proof covers the hand-translated models. There is no proof that the translation is correct. The liveness defects are only discussed. The manual effort is not given. - how much was read: most of it read - source; notes: Moderately Complex Paxos Made Simple: High-Level Executable Specificat - record id: `moderately-complex-paxos-made-2019` - Formal Verification of Usage Control Models: A Case Study of UseCON Using TLA+ (2018) - what: # Formal Verification of Usage Control Models: A Case Study of UseCON Using TLA+ (authored by agents unless marked 🧑) Antonios Gouglidis, Christos Grompanopoulos and Anastasia Mavridou, MeTRiD 2018, EPTCS 272, pp.52–64, DOI 10.4204/EPTCS.272.5. The paper specifies concurrent usage-control lifecycles and reports finite TLC checks for pre-authorisation and ongoing-authorisation models. The authors claim correctness for “≤ 12 uses”; Table 2 gives measurements at 2, 8 and 12. These are bounded model results, not a proof for arbitrary populations or an implementation. Printed temporal formulas also have substantial specification issues that cannot be reconciled with the reported checks without the executable models (Q0,Q7–Q10). ## Reading and evidence Read all 13 pages, sections 1–5 and all eleven references. Inspected four figures, two tables and all displayed definitions/actions/formulas. Viewed eight individual renders, pages 5–12, including enlarged printed safety/liveness formulas and Table 2. The inventory, official arXiv and official EPTCS PDFs are byte-identical. Formula symbols lost in extracted text were checked against the renders. No executable TLA+ module, TLC configuration or run log was acquired. Section 2 says “see TLA+ specification in [1]”, but reference [1] is Microsoft's TLA+ Tools page, not an identified UseCON model archive (Q1,Q11). The saved PDF's active links point to tools, cited publications and license information. This is a bounded artifact-location finding, not a claim that no source can exist elsewhere. No author code/model executed or imported, no TLC/compiler/checker/solver run and no serialized scientific results loaded. ## Model and concrete cases UseCON adds a use entity to subjects, objects and actions. A use records a request's subject/action/object identifiers, lifecycle status and mutable use attributes. U, the set of use records, is the only declared state variable. Subject/object/action records and their identifiers are treated as constants; administrative changes to their attributes are outside the modeled operation. Initially U is empty. Policy-related history can be read through existing uses (Q1,Q2). Direct policies use attributes of the requesting subject, object and action. Indirect policies select a related entity using CHOOSE; the example checks whether a child's father has premium membership. Complex indirect policies aggregate selected entities; the example sums a customer's account balances. Correct interpretation of CHOOSE requires an appropriate selected entity; existence/uniqueness conditions are not developed as part of these examples. The paper illustrates expressiveness but leaves verification of complex and interacting ongoing policies as future work (Q3,Q10). Induced use attributes come from participating entities, such as a service price. Observed attributes require an external observation, such as the system time when access is activated. Update procedures change status and other use attributes together. SystemTime is illustrative rather than a concrete bounded clock model. Administrators must supply policy-specific procedures and finite domains for any TLC instance (Q2–Q4; reader finite-model qualification). The pre-authorisation lifecycle is initial→requested→activated→completed or requested→denied. Request and Complete are subject actions; preEvaluate is a system action. Next is their disjunction. Request nondeterministically chooses a subject/action/object tuple absent from existing records. Because terminal records remain in U and the uniqueness test uses only that tuple, the printed transition rules do not permit requesting the same tuple again in that model lifetime. This restricts the workload compared with repeated real-world accesses (Q5; reader inference). For ongoing authorisation, Request is followed by Activate without a prior policy test. onEvaluate keeps the use active and updates attributes if the policy holds, or changes it to stopped otherwise; Complete changes active to completed. Next includes all four actions. The evaluation interval is explicitly left to implementation. Thus the model does not establish a maximum revocation delay or that a policy is continuously true between evaluations (Q6). All update actions are assumed atomic. The authors justify this with a central policy decision point and the absence of updates to subject/object attributes. Concurrent operations are represented by interleaving these whole actions; message loss, replication, distributed transactions, crashes and partial updates are not modeled here. There is no implementation-refinement proof showing that a deployed access-control service realizes these atomic steps (Q4–Q6). ## Intended properties and printed formulas TypeCorrectness is U⊆Uses, where Uses is a finite-domain record type. Table 1 lists forbidden status changes and intended progress. Pre-authorisation forbids leaving completed or denied and returning from activated to requested/denied; it expects requested uses eventually to activate or be denied, and activated uses eventually to complete. Ongoing authorisation similarly forbids leaving completed/stopped or returning from activated to requested; it expects requested uses to activate and eventually complete or stop (Q7). The printed safety formula is □(∃u∈U: u.st="completed" ⇒ □(u.st≠"requested")), where □ means always. This is existential, although the prose says the condition applies to all uses. With the natural quantifier scope shown by the parentheses, the existential is false when U is empty; Init explicitly sets U={} so the formula fails in the initial state. Further, u denotes a bound record value. The inner u.st does not look up that use's changing record in future U states. A completed bound record retains its completed field even if U later contains a replacement record with the same identifiers and a different status. These are semantic issues in the printed formula, not an executed TLC counterexample (Q4,Q7; reader interpretation). The liveness formula is ∀u∈U: u.st="activated" ↝ u.st="completed", where ↝ means that whenever its left condition holds, the right condition eventually holds. U is quantified at the point where this formula is evaluated. As a top-level property evaluated initially, it is vacuously true because initial U is empty. If evaluated at a later state with an active record, that bound record value does not evolve into the completed record. Adding an outer □ alone would not repair this identity/update issue; the property needs stable identifiers and a state-dependent lookup (Q2,Q4,Q8; reader interpretation). The paper supplies Init and Next but no complete temporal specification or fairness assumptions. A disjunction of enabled actions alone does not require an enabled completion or policy evaluation eventually to be selected. In the ongoing model, an active use can remain active through onEvaluate repeatedly while completion is postponed. If stuttering is allowed by the full specification, it gives another way to postpone progress. Scheduling/fairness and policy assumptions must therefore be explicit for an eventual-completion theorem. No such theorem can be recovered from the displayed transitions alone (Q5–Q8; reader separating behavior). There are also printed notation inconsistencies: action guards and createUse use state while other definitions and updates use st; record definitions and extra braces in the ongoing actions are not a complete executable module. These discrepancies could differ from the authors' actual checked models. They prevent treating the printed listings as independently validated source. The reported no-error checks and the initial-state safety failure cannot both refer literally to the same displayed property under standard quantifier interpretation; the missing modules/configurations leave the correspondence unresolved (Q2,Q5–Q9). ## Measurements and their scope The authors use TLA+ Tool 1.5.6 and TLC 2.12 on a mid-2014 MacBook Pro, OS X El Capitan, 2.5GHz Intel Core i7 and 16GB 1600MHz DDR3. Table 2 contains six configurations. Each tuple below gives uses; reported diameter; generated states; distinct states; deadlock-check seconds; safety/liveness seconds (Q9). - Pre: 2;6;21;12;1;3 - Pre: 8;23;277,969;16,832;2;5 - Pre: 12;31;45,533,665;560,128;53;138 - Ongoing: 2;7;33;16;1;3 - Ongoing: 8;25;367,873;23,808;2;7 - Ongoing: 12;37;79,112,449;1,224,704;118;241 All 36 numeric cells were transcribed from the rendered table. Diameter is retained as the paper's reported label; no TLC statistics were independently regenerated. State-count growth and interrupted larger runs demonstrate an exploration limit for this setup. Table 2 does not document every integer population below 12, all policy choices or all attribute domains. The exact subject/object/action split, policy instances, bounds and fairness configuration are not supplied in enough detail to reproduce these counts (Q9). The paper calls the first experiment deadlock-freedom checking, then describes TLC stopping at a deadlock after every pre-authorisation use is denied or completed. Its conclusion says “The trace of the deadlock errors verifies that the defined specifications of the system operate correctly.” A permitted terminal state and an unexpected stuck state need separate acceptance conditions. These traces can support an intended-termination inspection, but their existence does not itself establish deadlock freedom, exhaustive termination of every behavior or the temporal formulas above (Q9,Q10). The authors report no safety/liveness errors for the bounded configurations. No model files, logs, certificates or reproduced checks establish which executable formulas produced that result. The appropriate record is therefore an author-reported bounded check, accompanied by the printed-formula issues. Larger populations were stopped for time; no cutoff theorem extends the small checks to arbitrary numbers of uses (Q9,Q10). ## Contribution to the topic This is a concrete TLA+/TLC case study of security-related lifecycle modeling, finite-state exploration cost and intended safety/progress distinctions. Its central-state atomic model is relevant to concurrent systems but does not verify a distributed protocol or a Rust/Verus implementation. No LLM specification/proof agent is evaluated. Complex interacting policy verification, composition with secBIP and arbitrary-population guarantees are future work (Q10). For evaluating generated specifications, the paper provides useful failure modes: temporal quantification over an initially empty set, confusing a fixed record with a changing entity, omitted fairness, treating a terminal deadlock as a general correctness argument, and incomplete correspondence between printed formulas and reported checker success. These are reader lessons from this paper, not author-demonstrated LLM failures. - limits: boundedatomiccentralmodel;constantentityrecords;policydomains/fairnessmissing;printedformulasnotverifiedexecutablemodels - how much was read: most of it read - source - record id: `formal-verification-of-usage-2018` - Ivy liveness-to-safety (2018) - title: Reducing liveness to safety in first-order logic - what: After a finite prefix of an execution, picks a finite projection of the state (the freeze point). Proves that no two later states can agree on that projection while all fairness obligations are met. This turns a progress proof into a safety proof about a finite bad pattern. Cases: ticket, alternating bit, TLB shootdown, Paxos, Multi-Paxos, Stoppable Paxos. - number: finding the invariants 'requires a sophisticated user'; conjecture counts for liveness are '2 - 4 times higher' than for safety (§7.2, pp.26:27-28) - number: six cases: ticket protocol, alternating bit, TLB shootdown, Paxos, Multi-Paxos, Stoppable Paxos - number: checking takes 3.7 to 219 seconds excluding invariant discovery - number: liveness needs two to four times as many invariant conjectures as safety (§7.2) - limits: The user still supplies the inductive invariants (facts preserved by every step). Humans applied the reduction by hand. Paxos assumes that the actions of a chosen leader or quorum are fair, that messages are eventually delivered, and that competing proposals eventually stop. Multi-Paxos and Stoppable Paxos cover one incarnation with fixed membership. The comparison counts invariants, not person-time. - how much was read: most of it read - source; notes: Ivy: dynamic abstraction and temporal prophecy - record id: `ivy-liveness-to-safety-2018` - Parametric Timed Benchmark Library (2018) - title: A benchmark library for parametric timed model checking - what: Introduces a public library of academic, industrial and difficult parametric timed model-checking benchmarks, classified by model structure, parameters, properties and tool performance. The library standardizes IMITATOR-format models and metadata, includes robustness and partial-result cases, and gives a selected performance table spanning circuits, protocols, scheduling, real-time systems and timed pattern matching. It is designed for fair comparison and collaborative extension rather than proposing a new checker. - number: First version: 34 benchmarks, 80 models and 122 properties - number: Table uses Intel i7-7500U at 2.70 GHz with 8 GiB; timeout is 300 s - number: Selected times range from 0.01 s (train-AHV93) and 0.04 s (Fischer/Flip-flop) to 248.35 s (BRP), with several T.O./? entries - limits: The initial collection uses IMITATOR syntax and selected existing models; table times are machine- and tool-specific, with 300-second timeouts and labels for unsolvable or human-solvable cases. Translation to Uppaal may lose parameter/stopwatch information, and the library is a benchmark resource rather than a completeness guarantee. - how much was read: most of it read - - record id: `parametric-timed-benchmark-library-2018` - Fonseca study (2017) - title: An Empirical Study on the Correctness of Formally Verified Distributed Systems - what: The authors tested IronFleet, Verdi and Chapar. Over eight months they found 16 bugs, all at the edges between the specification, the tools and the unverified code. None were in the verified protocols. - number: 16 bugs; the PK toolkit automated detection of 13 - number: disabling deduplication via seven implementation lines still passed IronFleet verification - number: 16 bugs found, none protocol-level; the PK testing toolchain detects 13 of the 16 - number: IronFleet request-deduplication patch: about seven implementation lines changed, still verified - limits: Finding no protocol bugs does not show that none existed. The source of the PK toolkit was not found. - how much was read: most of it read - source; notes: Specification errors and auditing; notes: Empirical distributed-systems study and PK - record id: `fonseca-study-2017` - Paxos Made EPR (2017) - title: Paxos Made EPR: Decidable Reasoning about Distributed Protocols - what: The paper transforms Paxos-family models so that checking a human-supplied proof falls in EPR. EPR is a first-order fragment where satisfiability is decidable (an algorithm always answers). It checks proofs. It does not discover invariants. - number: Six variants: Paxos, Multi-Paxos, Vertical Paxos, Fast Paxos, Flexible Paxos, Stoppable Paxos - number: Verified six protocol variants with EPR transformations - number: Z3 experiments repeated 10 times, 300s timeout, Linux Core-i7 1.8GHz - number: EPR verification conditions solved in a few seconds with no timeouts; semi-bounded runtimes grew with rounds; unbounded Z3 diverged - limits: Z3 did not terminate on unbounded checks of the original models. The URL of the original artifact returns 404, and the review notes found no verified copy. - how much was read: most of it read - source; notes: Paxos Made EPR - record id: `paxos-made-epr-2017` - Who Guards the Guards? Formal Validation of the Arm v8-M Architecture Specification (2017) - what: The authors build a higher-level account of the security of the Arm v8-M architecture, which processor designers can review. They then formally compare it with the detailed processor spec. They check both forbidden access and intended permitted access. - number: twelve bugs found (including two security bugs) - limits: The trusted secondary specification still needs validation by experts. - how much was read: most of it read - source; notes: Specification errors and auditing; notes: Appendix B: Concrete systems and theorem scope - record id: `who-guards-the-guards-2017` - COS (Cloud Outage Study) (2016) - title: Why Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages - what: The authors read 1247 news items and public post-mortems about 597 unplanned outages of 32 internet services (2009-2015), tag each by root cause, impact and fix, and report what causes outages and why redundancy still fails. The data set is released as COS DB. - number: 597 outages, 1247 links, 3249 tags; 32 services; 2009-2015 - number: Root causes among known ones (355 of 597 unknown): upgrade 16%, network 15%, bugs 15%, config 10%, load 9%, cross-service 8%, power 6%, security 5%, human 4%, storage 4% - number: 78% of services (25) averaged less than 99.9% uptime; 6% (2) less than 99% - number: Impacts: full outage 59%, essential operation failure 22%, performance 14%, data loss 2% - limits: Only publicly reported outages, skewed to popular services; root cause known for only about 40% of outages. The paper does not give fixes or any formal method. Tags come from manual reading by at least four authors. - how much was read: most of it read - source; notes: ## Why Does the Cloud Stop Computing? (2016) - record id: `cos-cloud-outage-study-2016` - Multi-Paxos TLAPS (2016) - title: Formal Verification of Multi-Paxos for Distributed Consensus - what: # Formal Verification of Multi-Paxos for Distributed Consensus (authored by agents unless marked 🧑) Saksham Chand, Yanhong A. Liu and Scott D. Stoller. The original paper appeared at FM 2016, pp 119–136, DOI 10.1007/978-3-319-48989-6_8. This record reads the 37-page corrected and extended arXiv:1606.01387v4, dated 11 November 2019, classified cs.DC on its first page. The inventory's 2016 year identifies the original work; it does not date the corrected version read here (Q1,Q45; arxiv.html). The contribution is a small TLA+ specification of Multi-Paxos and its preemption extension, an author-reported TLAPS proof of per-slot agreement, and concrete strategies for managing the much larger proof. Two qualifications matter. The authors explicitly retract the earlier completeness claim: “The claim of a complete proof in [7] was incorrect” (§1 p2, Q3), explaining a PTL backend bug and their replacement type-invariant proof. Separately, this read finds inconsistent maximum axioms in the printed appendix and pinned proof files as written (Appendix C p29, Q49,Q59,Q63). Thus the record preserves the corrected checking report but does not treat checker acceptance under those retained assumptions as independent certification of the assumption-free theorem. This is a mathematical assumption problem, not an observed execution failure or a counterexample to Multi-Paxos agreement. ## Reading and evidence scope Read all 37 PDF pages in extracted text and rendered images: sections 1–7, all six figures, both numeric tables, all 26 numbered equations, acknowledgments, all 63 reference entries, and Appendices A–C containing the specification, invariants and hierarchical proof. The figures are algorithm/proof listings and comparisons, not statistical plots. The two PDFs, one from the local inventory and one freshly fetched from arXiv, are byte-identical. The local PDF is the corrected v4, not the 2016 proceedings text. Fetched the author-linked DistAlgo/proofs repository's metadata, commit and recursive tree. The pin is fb4a727ed8fab39197d9a20ed066d21abbb83573, last commit 4 November 2019. Read its full README and selected plaintext specification, assumption, type-invariant and final-theorem regions of the improved Multi-Paxos/preemption modules, plus the historical FM16 TypeOK regions. The README distinguishes these versions and says improved proofs were checked with TLAPS 1.5.6 (Q56). Downloaded Basic Paxos baseline and historical modules provide retained source context; this is not a whole-repository or whole-module audit. The explicit excerpt/range inventory is in source-scope.json. Also read the complete official issue 40 body describing the coalescing bug (Q66); metadata says opened 1 April 2017 and closed 17 May 2018. Closing an issue does not identify the fix commit or certify every installed tool version. No author code, model, TLAPS, TLC, SMT solver or proof assistant was executed or imported. No serialized data/model/checkpoint was loaded. Reader checks use file hashes, literal source slices, arithmetic and symbolic reasoning. Raw Git files retain their bytes; .lf.txt files are identified newline-normalized derivatives for character and LF-line quote pointers. The PDF extraction renders □ as 2, primes as 0 and some negated existential/membership symbols unusually; rendered pages were used to interpret the formulas. quotes.json retains extraction bytes, including such artifacts. ## The model and the exact theorem The system has proposer, acceptor and learner roles, which may be colocated. Constants describe proposers P, acceptors A, quorum family Q and possible values V. Ballots B and slots S are natural numbers. Membership and the quorum family are fixed: dynamic reconfiguration is excluded. The sole stated quorum condition is Q ⊆ SUBSET A and pairwise nonempty intersection. Majority, tree or matrix quorums can instantiate it once that condition is established; the paper does not construct or evaluate those alternatives (Q6,Q13,Q19). The state contains the set msgs of every message ever sent, proposer ballots pBal, each acceptor's highest seen ballot aBal, and aVoted, a set of ballot-slot-value records retaining the highest voted ballot for each slot. Init uses no messages/votes, proposer ballot 0 and acceptor ballot −1. Next chooses one proposer or acceptor phase. Spec = Init ∧ □[Next]_vars permits an action or stuttering at each step (Q13–Q19). The messages-as-history abstraction lets processes select any previously sent message, repeatedly and in any order. An unhandled message models arbitrary delay or loss; repeated handling models duplication. Crashes are modeled by not scheduling the crashed process, and recovery resumes its retained state. The authors require “all state (vars) is stored in stable storage” (§3.3 p11, Q20). The model has no Byzantine message fabrication, volatile-state reset, implementation of durable storage, or explicit network delivery queue. This is a safety abstraction of benign failures. There are no fairness or eventual-delivery clauses in the displayed Spec. Repeating Init forever is a valid behavior through UNCHANGED vars and never chooses a value. This reader witness demonstrates the absence of decision progress, without challenging agreement. A quorum of functioning processes and delivery assumptions would matter for liveness; no such liveness theorem is established here (Q9,Q15,Q19; math-witness.json). The target is Spec ⇒ □Safe, where Safe means any two chosen values for one slot are equal. The paper identifies C1 proposed-value validity, C2 agreement and C3 learner correctness, then says C1/C3 are excluded from the formal proof; §3.3 and §4.8 discuss them informally (Q5,Q22,Q33). There are no explicit learner variables/actions. Validity means a chosen slot/value occurred in a protocol 2a proposal, not that a value came from an external client or satisfied an application invariant. Equations 24/26 use sent although the state variable is msgs; the record treats this as the printed informal derivation, not a checked new theorem. The precise Chosen definition used in the proof is ∃b ∈ B: ∃Q ∈ Q: ∀a ∈ Q: VotedForIn(a,b,s,v). VotedForIn witnesses a sent 2b message with that ballot and slot/value. The introductory equations 3/4 place ∃b inside ∀a, allowing different ballots for different acceptors. Those predicates differ; use equations 10–12 and Appendix B for the actual theorem (Q6,Q23,Q48). A separating predicate valuation in math-witness.json makes the difference explicit, without claiming a reachable protocol counterexample. ## Minimal algorithm changes Basic Paxos carries one latest accepted ballot/value pair. Multi-Paxos carries a set of ballot-slot-value records in each promise. Phase1a sends a ballot. Phase1b responds only to a strictly higher ballot, returning its voted-record set and recording the promise. Phase2a waits for same-ballot promise replies covering a quorum and sends at most one 2a message per ballot. It selects the maximum accepted ballot separately per slot, retaining that slot's value, and adds new slot/value pairs. Phase2b accepts a ballot at least its promise, sends a 2b reply, and replaces voted records only for the slots in that proposal (Figures 2–5, Q14–Q18). MaxBSV filters the voted records to maximal ballots per slot; MaxSV drops ballots; UnusedS finds absent slots; NewSV chooses a set of unused-slot/value pairs with no duplicate slot carrying different values; PropSV combines old and new pairs. Ballots are shared across slots, and old unfinished slots can be reused. The authors explain that this is more involved than separately running independent Basic Paxos instances (Q15,Q17). NewSV has no explicit nonemptiness, finiteness or client-input requirement. The empty set satisfies its uniqueness predicate. CHOOSE denotes a fixed mathematical choice for the same argument, not a newly sampled nondeterministic result each time. The formulas are useful abstractions for agreement, but do not by themselves ensure an executable finite batch or useful new work. Finite msgs does not imply finite message payloads (Q17,Q47; math-witness.json). Preemption adds an addressed reply containing an acceptor's highest ballot when it rejects a proposer ballot. A proposer receiving a strictly higher preempt ballot updates pBal above it. Phase1a then sends pBal, and the two acceptor actions have extra rejection branches (Figure 6/Appendix A, Q34,Q35,Q47). In Phase1b equality also enters the else branch, although the prose often says lower ballot. The proposer's strict-higher guard prevents equality from raising pBal. Two source differences constrain exact correspondence. The printed NewBal requires only b>b2; the pinned NewBallot also excludes ballots already used by a 1a message, matching §5.1's fresh-ballot explanation. Conversely, the pinned non-preemption Phase1a omits the no-prior-1a guard displayed in Figure 2. Both retain the global no-prior-2a-per-ballot guard. These differences are documented, not silently normalized or labeled agreement violations (Q14,Q35,Q36,Q47,Q57,Q62). Actual software must allocate ballots, for example ordered counter/process-identifier pairs, rather than inspect a global history oracle. Reconfiguration, message/state reduction and failure detection are omitted. A 1b reply can contain votes for every previously used slot. There is no verified Rust implementation, code refinement or operational throughput experiment in this paper (Q21). ## Invariants, decomposition and final argument Inv conjoins TypeOK, AccInv and MsgInv. TypeOK constrains every message/function/record field; the corrected preemption version additionally requires finite msgs. AccInv relates each acceptor's promise and current voted records to the historical 2b votes: stored records were actually voted, every historical vote is covered by a same-slot record at least as high, and no vote lies above the current same-slot maximum (Q25,Q26,Q48). MsgInv1b records the sender's promise, actual historical votes in its returned set, and absence of votes between that slot's stored maximum and the promised ballot. MsgInv2a says every proposal is SafeAt its ballot, a message has at most one value per slot, and there is only one 2a message per ballot. MsgInv2b ties each vote to a matching 2a message and bounds its ballot by the acceptor's highest seen ballot (Q27,Q28). SafeAt(b,s,v) says that for every lower ballot, some quorum consists of acceptors that voted v or have promised above that ballot and never voted there. WontVoteIn expresses the latter alternative. VotedInv carries SafeAt from proposals into votes; VotedOnce states same-ballot/same-slot votes have the same value, even from different acceptors. Their role is concrete: at equal chosen ballots use VotedOnce; at unequal ballots intersect the chosen quorum with the SafeAt quorum. The common acceptor's old vote excludes WontVoteIn, leaving equal values (Q24,Q50,Q55). The proof first establishes Init ⇒ Inv and Inv ∧ [Next]_vars ⇒ Inv′, with separate action cases and an explicit unchanged-state case. PTL then derives Spec ⇒ □Inv and transfers Inv ⇒ Safe to the temporal Safety theorem (Q29,Q52–Q55). The final temporal QED is present; its presence still does not prove the assumed mathematics consistent. The reusable method separates unaffected old elements from each action's new message or voted-record increment. Invariance lemmas preserve old VotedForIn/SafeAt facts; the new element is proved from the sending action and local invariants. Conjuncts and actions form a mechanical decomposition, while difficult bounds and witnesses require manual facts. The paper illustrates this fully for Phase1b and MsgInv and shows the longer Phase2a proof in Appendix C (Q30,Q31,Q51,Q54). Proof syntax matters to performance. A common acceptor voting both values in the equal-ballot case is supplied explicitly; the authors report about 90 seconds reduced to a few milliseconds, calling it a 10,000× local speedup. Replacing direct record membership by an existential record with field equalities made another proof proceed after the original stalled for one or two minutes. These are local development observations, with no repeated-run distribution or uniform ablation (Q32). Tuple access is a concrete proof-engineering choice, not evidence of an algorithm runtime speedup. ## The disclosed checker bug and the remaining assumption problem The authors report that historical PTL coalescing could “prove” a primed state formula from its unprimed assumption outside a valid temporal context. It incorrectly accepted TypeOK′ under TypeOK ∧ [Next]_vars. The official issue's example falsely preserves xyz=0 while Next permits increasing xyz. Choosing increment 1 from xyz=0 gives xyz′=1, refuting that preservation claim symbolically (Q36,Q66). Historical FM16 modules retain PTL at primed message-type leaves; improved excerpts instead give explicit action proofs (Q60,Q64,Q65). The repair proves type preservation for all phases. Preemption needs a fresh larger natural ballot to exist. With finite msgs, the used 1a ballots form a finite set T; max(T∪{received ballot})+1 supplies a larger unused witness. The authors added IsFiniteSet(msgs), proved preservation under message addition and supplied this witness. They report initial TypeOK proof growth 54→93 lines and 75→138 lines for the two variants; subsequent improvements reduce those to 54 and 77 (Q36,Q37,Q52). A separate issue remains in the retained text. Appendix C calls two axioms “straightforward consequences” but writes MaxInSet as ∀S∈SUBSET N: Max(S)∈S (Q49). Substitute S=∅: it asserts Max(∅)∈∅, false for every CHOOSE result. The pinned source writes the domain as (SUBSET Nat) \\ {}, which subtracts no elements and still includes ∅ (Q59,Q63). Removing the empty set as an element would require {{}}. That change alone would still leave unbounded infinite subsets: applying MaxInSet and MaxOnNat to Nat gives a natural maximum k, but k+1 is a larger natural. A sound maximum lemma needs nonempty finite or bounded domains and proof that each use meets the restriction. These are static mathematical witnesses to inconsistent assumptions as written, not TLAPS runs, discovered new protocol executions, or an assertion that every leaf exploited inconsistency. The retained files cannot alone support an assumption-free machine-certification claim. Agreement may be true and the intended invariant argument may be repairable, but repairing and replaying it is outside this literature read. The disclosed PTL repair does not address these maximum axioms. This distinction is particularly useful when evaluating automated proof agents: green obligations and completed syntax must be accompanied by an audit of explicit axioms and the actual target. ## Evaluation and quantitative evidence The evaluation compares Basic Paxos, Multi-Paxos and Multi-Paxos with Preemption, plus older incomplete and corrected/improved proof variants. It measures text size, proof-tree structure, lemma reuse, obligations and proof-checking cost, not network performance or application throughput. Table 2 specifies a four-core Intel i7-4720HQ at 2.6 GHz, 16 GB RAM, 64-bit Ubuntu 17.10 and TLAPS 1.5.6. Earlier elapsed times from 1.5.3 are marked separately. No raw timing logs, repeat counts, variance or matched cross-tool benchmark are provided (Q40–Q42). The corrected new rows report specification lines excluding comments of 52, 55 and 74; proof lines 310, 787 and 831; obligations 239, 779 and 825; total CPU seconds 40, 175 and 180; total elapsed seconds 14, 51 and 90. CPU time and elapsed time are distinct; the total CPU values are close to the headline “about 3 minutes,” whereas the elapsed values are 51/90 seconds. Adding Multi-Paxos slots costs only three specification lines but 477 proof lines and 540 obligations; adding preemption costs 19 specification lines, 44 proof lines and 46 obligations. The per-slot data structure creates most of the proof expansion (Q40,Q41). The new proof trees have maximum level 10 and degree 17, against 7 and 3 for Basic Paxos. Multi-Paxos/preemption use five/six invariance lemmas 32/35 times and three proofs using set increments each. §4.6's five lemmas used 27 times describes an earlier count; Table 2 provides the updated reuse counts (Q31,Q41). Reflowed appendix counts, including 528 proof lines excluding comments, are layout alternatives, not another method or independently smaller proof. The three improvements factor message predicates, merge similar unchanged-state cases and remove leaf scaffolding TLAPS can establish automatically. Reported reductions for preemption include 75 lines from predicate factoring, 27 from merged cases, 110 from about 25 unnecessary-obligation instances and 61 from 22 TypeOK instances. The full repaired proofs shrink 1049→787 and 1117→831 lines; those are 25%/26% reductions from the repaired versions. Old incomplete rows marked ^^ are not valid proof-performance baselines (Q3,Q37–Q41). Table 1 gives improved TypeOK elapsed times 8/25 seconds; Table 2 gives 7/24. Their old preemption TypeOK elapsed entries also differ, 21 versus 22. Preserve these printed differences rather than inventing a run-variance explanation. numeric_tables.json retains every metric, alternate count and footnote marker. The report that adding preemption took less than an hour explicitly excludes TypeOK; it cannot be used as total authoring time for the corrected proof (Q34,Q37,Q41). ## Relationship to the requested research topics This is directly useful for TLA+ and formal verification of distributed protocols: concrete fault abstraction, per-slot agreement, quorum reasoning, stronger invariants, hierarchical proof feedback, old/new-set decomposition and sensitivity to formula representation. Its strongest evaluation lesson is the explicit correction history, reinforced by the separate maximum-assumption findings in this read. The work contains no Verus, verified Rust or LLM-agent experiment. Transfer to those topics is a reader proposal: preserve the exact property and fault model; measure changes in executable specification separately from proof scaffolding; give agents small action/conjunct obligations with explicit contexts; distinguish finite-instance testing from parameterized proof; and check axioms, omitted obligations, tool versions and source correspondence before reporting success. No agent success rate or Rust implementation result follows from this paper. Related-work proof-size comparisons with IronFleet, Verdi, EventML/Nuprl, EPR and other systems are author descriptions of differently scoped work, not controlled evidence that TLAPS is uniformly faster or requires fewer lines. The separate executable-program work in reference 45 reportedly found a safety violation missed by testing/model checking, but that example is cited rather than reproduced and verified here (Q43,Q44). Those leads can be followed as separate full reads. - number: Safety proof for Multi-Paxos - number: Proof strategies build on Basic Paxos - limits: Fixedmembership,intersectingquorums,naturalballots/slots,historicalmessages,stableallstateacrosscrashes,unfairstuttering. AgreementC2only; noformallyprovedC1/C3,liveness,clientvalidity,finitepayloadorimplementationrefinement. AuthorscorrectpriorPTLgap;readerfindsretainedmaximumaxiomsinconsistentaswritten;notindependentassumption-freesafetycertification. Noauthorcodeexecution. - how much was read: most of it read - source - record id: `multi-paxos-tlaps-2016` - A Messy State of the Union: Taming the Composite State Machines of TLS (2015) - what: The authors test TLS libraries for state-machine bugs by generating traces that deviate from the allowed behavior. They also write a replacement C state machine for OpenSSL and prove in Frama-C that it only allows traces from a hand-written list of allowed traces. A Coq proof shows that two sides with equal handshake logs are in equal states. - number: 8 libraries tested, flaws in 6, ten attacks including eight impersonation attacks - number: replacement about 750 C lines; invariant spec about 460 lines; annotations about 900 lines - number: about 2 minutes per function, 30 minutes total - number: Eight libraries tested, serious flaws in six, ten attacks including eight impersonation attacks - limits: The work proves nothing about integrity or confidentiality. It does not prove that the system eventually makes progress. It covers one component, not the whole library. The human effort was not timed. The authors read the 2015 version, not the CACM version. - how much was read: most of it read - source; notes: A Messy State of the Union: manual specification and annotations, auto; notes: A Messy State of the Union: manual specification and annotations, auto - record id: `a-messy-state-of-2015` - IronFleet (2015) - title: IronFleet: Proving Practical Distributed Systems Correct - what: It proves a replicated state machine (IronRSL) and a sharded key-value store (IronKV) correct, down to Dafny code. The liveness proofs are built by chaining intermediate progress conditions. - number: 1,400 specification, 5,114 implementation, 39,253 proof lines; 395 minutes to check - number: IronRSL liveness: 7,869 proof lines, 115 minutes; IronKV liveness: 2,093 lines, 23 minutes - number: about 3.7 person-years - number: implementation-level proof/code ratio 3.6 - limits: The top-level spec, event loop, Dafny, runtime, OS and hardware are trusted. IronRSL liveness needs eventual timing bounds and a live quorum. The reduction argument is on paper. - how much was read: most of it read - source; notes: S8 - record id: `ironfleet-2015` - Meeting a Powertrain Verification Challenge (2015) - what: Reports an extension of C2E2 that computes local trajectory-divergence bounds automatically and applies bounded-time reachability to a polynomial powertrain model. For two selected driver-behavior families, seven of ten printed properties are satisfied and three stricter variants are refuted, with50–53validated simulations for successes and2–4for failures. This is a model-verification case study; correctness/relative-completeness theory and detailed local-bound algorithm are delegated to earlier papers/report, not machine-checked here. Explains validated simulation plus discrepancy bloating, iterative initial-region refinement, Jacobian symmetric-part eigenvalue bounds, local coordinate changes and inactive-state reduction. Shows how selected STL obligations become timer-based safety requirements and preserves the limits of closed models and selected switching signals. Full result table exposes both positive and deliberately stronger negative specifications, with precise timings and hardware. Input schedules, initial radii, numerical validation and deferred proof assumptions prevent independently reproducing or generalizing the reported verification. - limits: Target is polynomial hybrid Model3 from the cited benchmark, not the full higher-fidelity engine suite. Four modes are startup, normal, power and sensor_fail. Continuous engine states are manifold pressure p, air/fuel ratio lambda, pressure estimate pe and integrator i. Introduction says five continuous variables; Figure1 lists these four plus throttle input, and an auxiliary timer is introduced for properties. The paper does not fully reconcile its state-dimension count; do not silently treat it as five autonomous engine states. Figure1 transitions: startup→normal when timer=Ts; normal→power when throttle>=70degrees; power→normal when throttle<=50degrees; normal→sensor_fail on sensorFail. Normal-mode equations are nonlinear polynomials in engine states with parameters ci, engine speed omega and throttle theta_in; Fc and mass flow are auxiliary expressions. Constants and other mode equations are not fully supplied, so this paper alone is not a complete numerical model specification. C2E2 handles closed automata only. Piecewise-constant driver inputs are closed by constructing a family of switching signals for each behavior of interest. Verification covers initial uncertainty and that selected family, not every possible driver, sensor-failure schedule or arbitrary open input. Authors choose two behavior sets said to visit all modes; exact signal ranges/schedules and branch-wise allocation to table rows are not printed. Initial uncertainty is a state-space ball. Its center, radius, exact cover and coordinate units are not supplied. Time horizon T=20 and startup Ts=9 are reported. Figure2 shows an illustrative trajectory/reachable envelope over startup/normal/power/normal; it does not by itself demonstrate sensor-fail coverage or reproduce all selected schedules. Validated simulation is defined as time-stamped regions enclosing the ODE solution on each interval with bounded diameter. A discrepancy function bounds distances between any compared trajectories and vanishes as initial points converge. These are mathematical obligations supporting bloated reachability; ordinary sampled trajectories without valid bounds would not suffice. Algorithm covers the initial region with delta-balls, validates a representative trajectory, bloats by discrepancy, then refines ambiguous balls. Disjointness from unsafe states establishes safety; violation needs a genuine unsafe execution or a certified enclosure contained in the unsafe region at a relevant time. Mere overlap of an overapproximation with unsafe states is inconclusive, not a counterexample. The terse interval/set exposition does not fully formalize this temporal decision rule. Soundness and relative completeness are attributed to the discrepancy properties and Theorem13 of ref6; full assumptions are not repeated. Vanishing enclosure error alone is not a claim that every boundary-touching model terminates or that safety is decided under all numerical conditions. Imported proof and its implementation correspondence have not been independently checked. Automatic local discrepancy uses Jacobian Jf and a Lipschitz constant Lf, a coarse short-time reachable enclosure, and an upper bound on the maximum eigenvalue of the symmetric Jacobian over that region. Rates may be negative. Full construction/correctness is deferred to ref10. Tool still requires a user-supplied Jacobian, so removal of discrepancy annotations is not removal of all mathematical input or proof obligations. Implementation uses Eigen for eigenvalues and interval arithmetic to maximize error-matrix norms. No exact arithmetic, outward-rounding contract, user-Jacobian validation or certified Eigen error bounds are described. These are uninspected implementation/trust boundaries, not demonstrated failures. Model translation from Simulink to the C2E2 automaton is not proved correct here. Local coordinate changes mitigate spurious positive divergence bounds for systems with complex eigenvalues, trading exponential-rate improvement against a multiplicative error factor. The coordinate-update horizon is tuned; no numeric setting or ablation is given. Startup/power reduction omits i from discrepancy computation only where i_dot=0 and i is absent from other right-hand sides; this is not unrestricted integrator elimination in normal mode. Properties reducible to bounded-time invariants use timers since trigger events. Here rise denotes throttle changing from0 to60. For rise, lambda stays within a reference band after eta and until zeta. Table settling intervals are open(eta,zeta), unlike some closed invariant intervals; boundary conventions matter. Arbitrary STL, unbounded liveness and path-integral properties are not covered; integral obligations were explicitly excluded. Reported parameters eta=eta_power=1, eta_s=.5, Ts=9,T=20,lambda_ref=14.7,lambda_power_ref=12.5,zeta=4. Seven positive rows use50or53simulations and615–718seconds; three negative rows use2or4simulations and24,43,45seconds. Simulation counts describe representatives/enclosure construction, not independent random tests or enumeration of every initial state. Experiments use an Intel Quad-Core i7,8GB RAM and Ubuntu11.10. No repeats, variance, parallelism settings, exact CPU model, solver tolerances or raw traces are printed. Faster negative rows are attributed to early exit, not a statistically established universal relation between satisfiable/violated-property runtime. Flow* is said to be unable to verify these models due to nonlinear complexity, but no version, resource limit, settings or comparison table is supplied. Treat as an author report, not a controlled head-to-head benchmark. First-verification priority is also an author knowledge-qualified claim. Tool/model link is http://publish.illinois.edu/c2e2-tool/powertrain-challenge/. Artifact contents, user Jacobian, initial set, switching family and returned witnesses were not opened/run. No machine-checked proof or independently reproduced verification result is established in this reading. Higher-fidelity variants with delay differential equations, actuation delays and lookup tables are future work and explicitly unsupported. No verified Rust/Verus, TLA+, LLM specification/proof agent or distributed-system implementation is evaluated. Methodological relevance is the separation between simulation testing and validated overapproximation, and between selected environmental assumptions and universal model coverage. - how much was read: most of it read - source - record id: `meeting-a-powertrain-verification-2015` - AWS TLA+ report (2014) - title: Use of Formal Methods at Amazon Web Services - what: Engineers describe using TLA+ and the TLC model checker on ten AWS designs. TLC is a tool that tries every reachable state of a design to look for bugs. Before checking design changes, they wrote down the failure modes and the environment. - number: ten systems, seven teams; useful results after two to three weeks of learning - number: S3 networking 804 PlusCal lines, two bugs; DynamoDB replication 939 TLA+ lines, three bugs; EBS 102 PlusCal lines, three bugs - number: one DynamoDB bug needed a 35-step counterexample - number: Engineers learn TLA+ and get useful results in 2 to 3 weeks - limits: The method checks designs, not code, and does not prove anything about code. A lock-free structure still had a bug where something should eventually happen but never does. TLC missed it because the engineers did not check that kind of property. - how much was read: most of it read - source; notes: S13 - record id: `aws-tla-report-2014` - Spin Paxos (2014) - title: Model Checking Paxos in Spin - what: Models Paxos in executable Promela with proposer, acceptor and learner roles, then uses Spin plus reduction theorems and code transformations to check finite instances. Counting guards and quorum transitions encode majority voting atomically, reducing interleavings. Formalizes safety as learner agreement, proves finite reductions for proposers/learners, and introduces atomic broadcast and counting-guard transformations. Experiments extrapolate quorum constraints and show ordered channel insertion nearly halves state space versus FIFO sends. - number: 2 proposers/2 acceptors: majority 1 violation, majority 2 exhaustive safety - number: 3 acceptors: majority 1 violation, majority 2 no violation; channel state stabilizes at size 6 - number: 4 acceptors: majority 3 correct; ordered-send model at 3 acceptors/channel 6 has 65,091 states versus 172,868 FIFO - number: 5 acceptors exhaustive exploration unmanageable; 16GB machine, Intel i7-4700HQ - limits: Finite-state bounded channels and process counts; no Byzantine faults; Paxos may diverge if proposers inject proposals indefinitely. Acceptor-bound cutoff remains conjectural. Bitstate hashing estimates large instances rather than exhaustive proof. - how much was read: most of it read - - record id: `spin-paxos-2014` - AMD-FP-Multiplier (2011) - title: Formal Verification of an Iterative Low-Power x86 Floating-Point Multiplier with Redundant Feedback - what: Reports production-level formal verification of an iterative low-power x86 floating-point multiplier with redundant carry-save feedback, x87/SSE formats and rounding, using ACL2 theorem proving plus model checking of scheduler properties. Translates Verilog RTL to ACL2, proves behavioral equivalence and bit-level arithmetic invariants, handles clock gating hierarchically, and reuses the proof effort for a second processor generation. The paper details carry tracking, sticky-bit rounding, pipeline constraints, and scheduler verification. - number: About 8,500 new custom ACL2 theorems were interactively developed - number: Certification took about 11 hours on one machine, or six hours with parallelization - number: Verification reused for two instances in a second processor core with little overhead - number: Multiplier latency is 2–5 cycles; array is 76 × (27+2) bits - limits: The report focuses on multiplication while division/square-root support is only discussed as context. ACL2 proofs require interactive theorem/hint development; model checking covers scheduler properties rather than the full arithmetic datapath. Production details and proprietary tooling constrain reproducibility. - how much was read: most of it read - - record id: `amd-fp-multiplier-2011` - Model-Based Trace-Checking (2011) - what: Replays observed traces of a distributed Java travel-booking application through B (ProB) and Promela (Spin) models. It checks invariants and a resource-allocation progress property on those particular traces. - number: six invariant clauses - number: progress bug: keeps trying Hotel1 when it is full but Hotel2 has a room - limits: It finds errors but cannot show there are none. The match between model and code is assumed from inspection. Tracing code was inserted by hand. No success rate, size or time is given. - how much was read: most of it read - source; notes: Model-Based Trace-Checking - record id: `model-based-trace-checking-2011` - D3S (2008) - title: D3S: Debugging Deployed Distributed Systems - what: Developers write C++ predicates over consistent global snapshots. D3S injects instrumentation into running programs and checks the predicates. It reports the state histories that violate them. - number: 5 deployed systems, results within one month - number: PacificA 67,263 code lines / 118 predicate lines, 3 bugs; MPS Paxos 6,993 / 50, 2 bugs - number: runtime overhead less than 8%, below 2% in search and BitTorrent cases - number: 5 deployed systems, PacificA 67,263 code lines / 118 predicate lines with 3 correctness bugs; MPS Paxos 6,993 / 50 with 2 bugs - limits: It checks observed runs and does not prove anything. Snapshot correctness depends on group membership. The window is finite, so it cannot check liveness over infinite runs. Developers must pick useful predicates. - how much was read: most of it read - source; notes: D3S: Debugging Deployed Distributed Systems; notes: D3S: Debugging Deployed Distributed Systems - record id: `d3s-2008` - Automated Repair of AI Code with Large Language Models and Formal Verification - what: Expands NeuroCodeBench by mutation to about 81k neural-network C/C++ programs, verifies memory safety with ESBMC, and prompts LLMs to repair detected defects. Compares prompt layouts and iterative repair, including verifier feedback and history variants. - number: NeuroCodeBench expanded to about 81,000 programs - number: Best non-iterative repair rate about 18%; iterative approach about 25% - number: Compilation and verification rates are compared across prompt and temperature variants - limits: Mutation-generated programs and ESBMC findings define the benchmark; successful compilation/verification does not establish broader functional correctness. Results vary by LLM, prompt, temperature and verifier context, and the report calls for more diverse models and experiments. - how much was read: most of it read - - record id: `automated-repair-of-ai` - Creating an LLM benchmark suite for specification inference - what: A proposed benchmark with four tasks: add memory-safety specs, strengthen them into specs of full behavior, repair wrong specs, and generate lemmas. Each task runs zero-shot, few-shot, or interactively with verifier feedback. The verifier is Viper. - limits: A proposal, not a result. It gives no metrics. - how much was read: most of it read - source - record id: `creating-an-llm-benchmark` - IronFleet repository - title: IronFleet code release - what: The official code release contains verified IronRSL and IronSHT implementations, with Dafny build and verification instructions. The README says the release proves safety and liveness requirements and reports performance competitive with a reference system. - limits: The README describes the release and commands; it does not give a reproducible run log here. - how much was read: most of it read - source - record id: `ironfleet-repository` - IvyBench - title: IvyBench: Collection of Distributed Protocol Verification Problems - what: IvyBench collects 54 safety-checking problems in Ivy, mypyvy and VMT formats: Paxos, TLA-derived commit examples, locks, elections, I4 cases, FOL-IC3 cases and two DistAI cases. - number: 54 safety checking problems: 10 Paxos, 5 TLA-derived, 14 ex, 7 I4, 16 mypyvy/FOL-IC3, 2 DistAI. - number: 54 problems: paxos 10, tla 5, ex 14, i4 7, mypyv 16, distai 2 - number: README says 54 safety checking problems; six advertised groups total 54, but multisig adds four files making 58 Ivy inputs - number: Group sizes: distai 2, ex 14, i4 7, multisig 4, mypyv 16, paxos 10, tla 5 - limits: The README lists benchmark membership, not pass rates or independently reproduced results. - how much was read: most of it read - source; notes: IC3PO and its shared ivybench collection - record id: `ivybench` - Perennial examples - title: Perennial example programs - what: The examples include an allocator, durable append-only inode, directory composition and a replicated disk block. The README emphasizes modular composition and recovery obligations. - limits: Examples are pedagogical and repository-specific; no deployment guarantee is claimed. - how much was read: most of it read - source - record id: `perennial-examples` - P3 (2026) - title: P3: Joint Program-and-Proof Planning for Verified Code Generation - what: P3 plans the implementation and the proof together before generating either. It also introduces Lean4Commit0, a benchmark built from real libraries. - number: Lean4Commit0: 108 libraries, 511 code and 519 theorem placeholders - number: also 189 Verina and 77 AlgoVeri tasks - number: gains of 4.6-11.2 points over the stronger baseline; cost and time down up to about 40% and 37% on difficult subsets - number: improves by 4.6-11.2 percentage points (as quoted) - limits: The specifications are provided. The LLM review during curation can miss flaws in the specs. - how much was read: parts read - source; notes: papers - record id: `p3-2026` - SV-COMP (2026) - title: Competition on Software Verification - (SV-COMP). - what: An annual competition. Verifiers run on fixed programs and properties (concurrency, termination). Each returns an answer, a witness (evidence for the answer) and CPU time. - number: 2026 scoring: correct FALSE +1, correct TRUE +2, UNKNOWN 0, incorrect FALSE -16, incorrect TRUE -32 - limits: Witnesses are not required for TRUE results in unsupported or demo categories. The specs are fixed, so it does not evaluate inventing specs. Termination differs from a service that keeps responding forever. - how much was read: parts read - source; notes: SV-COMP - record id: `sv-comp-2026` - SWE-Proof (2026) - title: SWE-Proof / BenchProofer - what: Takes SWE-bench Verified issues, given as natural language plus a repository, with Nagini, Velvet and Lean backends. Three adversarial LLM auditors check the specs that agents write. It is the only benchmark the review notes found that starts from a vague task in a real repository. - number: 500 instances - number: A correct formal spec lifts resolution from 85% to 95%; only 56% of agent-written specs pass the audit - number: Specs from unresolved instances fail audit 92% of the time vs 51% for resolved - number: 26.8% of test-passing patches overturned by the audit; Lean faithfulness failure 37.2% - how much was read: parts read - - record id: `swe-proof-2026` - Veil blog (Sergey) (2026) - title: Verifying Distributed Protocols in Veil - what: Reports Claude Code finding agreement and validity invariants for FloodSet, using feedback from the checker. The faulty model lumped all crashes together with the broadcast and left out partial delivery, which made agreement hold trivially. - limits: A case study by a single author, not a comparative score. - how much was read: parts read - source; notes: papers inspected; notes: Veil: proofs in Lean with automatic first-order checks - record id: `veil-blog-sergey-2026` - VeriEquivBench (2026) - what: Given a natural-language problem, the task is to produce Dafny code and a spec. Dafny then proves that the code implies the spec and the spec implies the code. - number: 2,389 problems - number: Claude-4-sonnet 75.81% on CloverBench but fails on all instances here at pass@4; GPT highest mutual-equivalence rate 2.65% - number: 10% of expert-written DafnySynthesis specs wrongly claimed as ground truth, another 18% found by their review - limits: A trivially simple matched pair still passes. - how much was read: parts read - source - record id: `veriequivbench-2026` - Verified Detection and Prevention of Concurrency Anomalies in Multi-Agent Large Language Model Systems (2026) - what: The paper writes four concurrency anomalies of shared memory between LLM agents in TLA+, checked with TLC. It verifies Rust detectors and runtimes in a Verus crate. - number: 274 Verus obligations (zero assume, zero admit) - number: 24 obligations for the spec-implementation chain; refinement obligations 31 (pessimistic), 18 (SSI projection) - number: 274 Verus proof obligations - limits: Nothing mechanical links the TLA+ and Verus parts. The TLAPS proofs only check that the texts agree, and the Verus refinements are against Verus state machines. It covers safety only, and the authors state that liveness is a gap. It has a single author and the results are self-reported. The notes read about 25% of the paper. - how much was read: parts read - source - record id: `verified-detection-and-prevention-2026` - 3SF model checking (2025) - title: Technical Report: Exploring Automatic Model-Checking of the Ethereum specification - what: The authors translate the Python spec of Ethereum's 3-slot-finality (3SF) consensus protocol into TLA+ and use the Apalache model checker to test Accountable Safety: if two conflicting chains are both finalized, at least one third of validators can be shown to have broken a voting rule. They then add hand-made abstractions and re-encode the problem in SMT (CVC5) and Alloy to check larger instances. - number: Direct translation (Spec2): checking Accountable Safety timed out after more than 40 hours (Table 1) - number: Alloy with Kissat: Accountable Safety holds for up to 6 blocks; e.g. 5 blocks, 25 checkpoints, 12 votes in 234 s (Table 5) - number: CVC5: verified up to 5 blocks with 5 checkpoints (1h 40m) or 3 blocks with 6 checkpoints (74h) (Table 4) - number: A 7-block Alloy run went over 16 days without finishing - limits: Only the finality gadget and only Accountable Safety, on small bounded instances (at most 7 checkpoints and 24 votes); every validator has stake 1. The hand-made abstractions are not proved equal to the Python spec. Appendix A (translation rules) and B (soundness proofs) were skimmed, not read closely. - how much was read: parts read - source; notes: ## Exploring Automatic Model-Checking of the Ethereum specification (2 - record id: `3sf-model-checking-2025` - Clock-SI TLA+ (2025) - title: Formal modelling and analysis of Clock-SI with TLA+ - what: A master's thesis that models the Clock-SI protocol (snapshot isolation for a partitioned database using loosely synchronized clocks) in TLA+ in two versions, a detailed one and a simpler one, and uses the TLC model checker to test that the protocol gives Snapshot Isolation, stated with a client-centric specification. - number: Abstract model, Snapshot Isolation check: 2 keys, 2 partitions, 2 txns 3 s; 3 txns 3 min 28 s; 3 keys, 2 partitions, 3 txns 11 h 23 min 20 s; 3 keys, 3 partitions, 3 txns timed out (Table 10) - number: Detailed model: 3 keys, 2 partitions, 2 txns 21 min; most larger cases timed out (Table 9) - number: Symmetry and view features cut one 2-key, 2-partition, 3-txn state-space run from 2 h 54 min to 2 min 2 s (Table 8) - number: Machine: i7-7700HQ, 16 GB RAM; each time is the average of 3 runs; maximum runtime 12 hours - limits: Bounded to at most 3 transactions, keys, partitions; the target of 3/3/3 was not reached. The abstract model removes the message system, so it does not cover the real messaging. Only the safety property (Snapshot Isolation) is checked; no liveness. Chapters 2-3 (background) and the model listings in 4.1-4.2 were skimmed. - how much was read: parts read - source; notes: ## Formal modelling and analysis of Clock-SI with TLA+ (2025) - record id: `clock-si-tla-2025` - Design and Modular Verification of Distributed Transactions in MongoDB (2025) - what: A TLA+ spec of MongoDB distributed transactions, with a modular spec of the interface. A follow-up blog generates conformance tests against WiredTiger. - number: TLA+ spec about 571 lines plus storage spec about 332 lines - number: 87,143 generated tests in about 40 minutes on a small finite model (2 keys, 2 transactions) - limits: It is tested only. Code is at https://github.com/mongodb-labs/vldb25-dist-txns. - how much was read: parts read - - record id: `design-and-modular-verification-2025` - Korg thesis (2025) - title: Verification and Attack Synthesis for Network Protocols - what: Proves properties of three network-protocol pieces with theorem provers (Karn's RTT algorithm and TCP's timeout rule in Ivy and ACL2s, and Go-Back-N efficiency in ACL2s), model-checks TCP, DCCP and SCTP handshakes in SPIN, then builds Korg, a tool that searches for attacker behaviour that breaks a given property. - number: Handshake properties: TCP 4 (phi1-phi4), SCTP 10 (gamma1-gamma10), DCCP has its own set - number: Korg asked for at most 10 attacks per property; default search depth 600,000, maximum 2,400,000 - number: TCP phi3 (peers do not get stuck) took 72 min to 223 min per attacker model; other TCP properties took seconds - number: SCTP: one patched-vs-unpatched check; with the CVE-2021-3772 patch, Korg finds no off-path attack, which proves the patch closes it for the model - limits: Models are small by design to avoid state-space explosion; the handshake models are not linked to the Ivy/ACL2s models. Korg handles one centralized attacker, not coordinated multi-process attackers, and is exact only for the model, not the real code. Models took long to write by hand. - how much was read: parts read - source; notes: ## Von Hippel thesis on protocol verification and attack synthesis (20 - record id: `korg-thesis-2025` - PolySemist (2025) - title: Accelerating Protocol Synthesis and Detecting Unrealizability with Interpretation Reduction. - what: A tool that fills in the missing guards and updates of a half-written TLA+ distributed protocol (a sketch) so the protocol meets a safety or liveness property. It loops: the TLC model checker finds a counterexample, the tool rules out every candidate that shares it, and it skips candidates that behave the same on the states seen so far. - number: 7 benchmark protocols: 2PC, consensus, decentralized lock, lock server, sharded KV store, two reconfigurable Raft variants - number: faster than the previous tool Scythe in 160 of 171 realizable problems; total time about 9,000 s vs 32,000 s on those both solve - number: whole distributed-lock protocol (6 expressions over 2 actions) synthesized in 125 s; Scythe timed out at 1 hour - number: detected 'no protocol exists' in 80 of 123 unrealizable problems vs 16 for Scythe (easy 46/50, medium 21/35, hard 13/38) - limits: A human supplies the sketch (state variables, actions, hole grammars) and the property; results hold only for the chosen finite parameters (e.g. number of nodes), and the all-parameters step is left to TLAPS proofs from prior work. Termination is guaranteed only when each hole grammar has finitely many expressions up to interpretation equivalence. - how much was read: parts read - source; notes: ## PolySemist: protocol synthesis by sketching (2025) - record id: `polysemist-2025` - Scimitar (2025) - title: Interpretable Safety Verification of Distributed Protocols by Inductive Proof Decomposition - what: Scimitar breaks a proof into a graph of lemmas and transitions. This exposes where guidance is needed. The target is Raft LogMatching, on a modified model of about 600 TLA+ lines. - number: 21 hours, 33 total lemma nodes - number: grammar has 173 predicates - number: 33 lemma nodes - number: 173 grammar predicates - limits: The draft the notes inspected has a different title from earlier and later records, and a placeholder conference header. One property of an edited model is not a complete verification of Raft. - how much was read: parts read - source; notes: papers inspected - record id: `scimitar-2025` - Terzian TLA+-to-Rust (2025) - title: A Tale of Two Refinements: LLM-assisted TLA+ to Rust - what: The report repeatedly refines TLA+ models and asks Gemini to implement an image-cache example as multithreaded Rust. Its history records a livelock (threads keep running but make no progress) and a premature termination. Both needed correction. - limits: This is a project report and is not peer reviewed. No Verus proof was found. The claim of "fully verified, spec-driven code generation" is only a possibility. - how much was read: parts read - source; notes: R01. A Tale of Two Refinements: LLM-assisted TLA+ to Rust - record id: `terzian-tla-to-rust-2025` - Vest (2025) - title: Vest: Verified, Secure, High-Performance Parsing and Serialization for Rust - what: Users describe a binary format, and Vest generates Rust parsers and serializers verified in Verus. It uses a reusable combinator library. - number: about 6K library lines, verified in under ten seconds - number: DSL compiler 5,807 Rust lines - number: format descriptions: Bitcoin about 67 lines, TLS 1.3 about 500, WebAssembly about 600 - limits: It proves that parse and serialize are inverses, plus memory safety. It does not prove consensus or handshake security. Timing side channels are not a Verus theorem. The DSL frontend is unverified. - how much was read: parts read - source; notes: Vest: Verified, Secure, High-Performance Parsing and Serialization for - record id: `vest-2025` - Kondo (2024) - title: Inductive Invariants That Spark Joy: Using Invariant Taxonomies to Streamline Distributed Protocol Proofs - what: Kondo derives inductive invariants for Dafny protocol proofs from the protocol's structure. This reduces the invariants the user must write. - number: Paxos: 418 lines, Kondo needs 20 user invariants, 777 total proof lines, 42.9 s - number: IronFleet 4,581 lines of Dafny and Verdi 50,000 lines of Coq for manual proofs - number: Paxos row: 631 model lines; 27 invariant clauses without Kondo vs 20 synchronous; 856 proof lines without Kondo vs 557 synchronous plus 220 modified - number: Table 1 p. 848: 10 systems (Echo Server, Ring Leader Election, Simplified Leader Election, Two-Phase Commit, Paxos, Flexible Paxos, Distributed Lock, ShardedKV, ShardedKV-Batched, Lock Server) - limits: Liveness proofs are beyond its scope. Line counts are not measured human hours. - how much was read: parts read - source; notes: papers inspected - record id: `kondo-2024` - SandTable (2024) - title: SandTable: Scalable Distributed System Model Checking with Specification-Level State Exploration - what: It model-checks a TLA+ spec quickly, then replays the bug traces in the code. A conformance checker replays model traces to find places where the spec and code differ. - number: 23 bugs, 18 new, 17 confirmed, 13 fixed - number: Half a person day to a few person days to fix modeling errors; two person weeks adapting ZooKeeper specs - number: 114x-2989x faster state exploration - number: abstract: 18 new bugs, 17 confirmed, 13 fixed (the note's sentence adds 'out of 23 total' garbled) - limits: It cannot guarantee that no differences remain between the spec and the implementation. - how much was read: parts read - source; notes: P10. SandTable: Scalable Distributed System Model Checking with Specif - record id: `sandtable-2024` - ByMC survey (2023) - title: Survey on Parameterized Verification with Threshold Automata and the Byzantine Model Checker - what: Protocols are modeled by counting messages and participants against thresholds. This lets the tool check safety, and some liveness, for any number of participants, under conditions like n>3t (n participants, at most t faulty). - number: Table 4: 13 asynchronous/randomized algorithms (FRB, STRB, ABA, NBACG, NBACR, CBC, CF1S, C1CS, BOSCO, Ben-Or, RABC, kSet, RS-BOSCO) - number: Table 6: 16 synchronous cases, 14 verified; HybridQueen and kSetOmit have no successful check - limits: People still supply the models and assumptions. Termination of randomized protocols holds only against adversaries that act in rounds. The Tendermint claim covers one round only. The synchronous checker is a separate prototype. - how much was read: parts read - source; notes: ByMC and threshold automata - record id: `bymc-survey-2023` - PGo (2023) - title: Compiling distributed system models with PGo - what: It compiles Modular PlusCal models into Go. - number: Table 2 (MPCal SLOC; glue SLOC; effort): lock service 67; 87; 2 person-days; Raft 771; 676; 22 person-days; distributed KV 256; 383; 3 person-days; primary-backup KV 420; 270; 10 person-days; state-based CRDT 160; 185; 10 person-days - number: monolithic RaftKV: 25 person-days, 758 model lines, 1099 glue lines - limits: It gives no guarantee about the compiler. - how much was read: parts read - source; notes: P07. Compiling Distributed System Models with PGo; notes: papers - record id: `pgo-2023` - FoundationDB (2021) - title: FoundationDB: A Distributed Unbundled Transactional Key Value Store - what: Runs the real database actors in a deterministic simulated environment. The simulation replaces the network, disk, clocks and randomness. This gives reproducible schedules and recovery tests after failures stop. - limits: Simulation needs assertions or an oracle to decide what is correct. Its coverage is not a refinement proof or a proof of progress at unbounded size. - how much was read: parts read - source; notes: P23. FoundationDB: A Distributed Unbundled Transactional Key Value Sto - record id: `foundationdb-2021` - Temporal prophecy (2021) - title: Temporal prophecy for proving temporal properties of infinite-state systems - what: Adds prophecy variables (recorded predictions about future execution) to the earlier abstraction method. This lets it keep distinctions that the abstraction lost, without strengthening the contract with the environment. - number: four cases: ticket protocol, alternating bit, hybrid reliable broadcast, TLB shootdown - number: user invariant conjectures 60, 70, 58, 102 - number: checking takes 7, 25, 25, 240 seconds - number: Figure 3 (model lines; conjectures; checker seconds): ticket with task queues 90; 60; 7; alternating bit 143; 70; 25; hybrid reliable broadcast 166; 58; 25; TLB shootdown 468; 102; 240 (checking time, not authoring time) - limits: Prophecy formulas are extracted automatically only from an invariant the user supplies. Choosing the witness is manual. No claim of completeness. - how much was read: parts read - source; notes: Ivy: dynamic abstraction and temporal prophecy; notes: additional temporal methods - record id: `temporal-prophecy-2021` - Elle (2020) - title: Elle: Inferring Isolation Anomalies from Experimental Observations - what: A test-time checker that runs random transactions against a database, records what clients saw, and rebuilds a dependency graph between transactions. A cycle in that graph is a proof that the database broke its claimed isolation level. - number: Found anomalies in all 4 databases tested: TiDB, YugaByte DB, FaunaDB, Dgraph - number: Checks histories of hundreds of thousands of transactions in tens of seconds; Knossos often fails after a few hundred - number: Linear time in history length - limits: Needs generated workloads where reads reveal the full write history (such as list append). Ignores predicates, so it cannot separate repeatable read from serializability. It finds bugs, not proof that a database is correct. - how much was read: parts read - source; notes: ## Elle isolation checker (2020) - record id: `elle-2020` - eXtreme Modelling (2020) - title: eXtreme Modelling in Practice - what: MongoDB compared two ways to link a model to code. One is model-based trace checking (MBTC), which checks whether logged runs fit the model. The other is model-based test-case generation (MBTCG). Trace checking of replication was impractical. For Realm Sync's operational-transformation rules, a Go generator turned TLC states into C++ tests. - number: Realm Sync table (§5.2): 795 TLA+ lines, 755 Go lines, 4 weeks total effort - number: Abandoned trace checking after 10 weeks - number: 4913 generated tests reached 100% branch coverage vs 21% handwritten and 92% AFL fuzzer - limits: Trace checking of replication struggled with the level of abstraction, concurrency, incomplete observations, and keeping code and model in sync. The generated tests cover local merge behavior, not the whole distributed product. - how much was read: parts read - source; notes: P04. eXtreme Modelling in Practice - record id: `extreme-modelling-2020` - Pretend Synchrony / Goolong (2019) - title: Pretend synchrony: synchronous verification of asynchronous distributed programs - what: Verifies Go distributed programs by automatically rewriting them into an equivalent synchronous program (messages become assignments), then checking ordinary verification conditions with the Z3 solver. Four case studies: two-phase commit, Raft leader election, single-decree Paxos, Multi-Paxos key-value store. - number: manual invariant annotations reduced by 6x versus Dafny - number: check time 1.73 s total vs 1455.84 s in Dafny for the three protocols - number: Paxos 504 lines of Go, 14 loop-invariant lines, 1.51 s; Dafny 72 invariant lines, 1141.35 s - number: Multi-Paxos KV: 847 Go lines, 1.64 s check, 118.5 req/ms, within 1.5-3x of unverified EPaxos code - limits: Safety only, no liveness. Needs structured loops, symmetric process ids, typed channels, pairwise communication and round non-interference; rules out Multi-Paxos stable-leader optimization, Stoppable Paxos and ring topologies such as Chord. - how much was read: parts read - source; notes: Pretend Synchrony (2019) - record id: `pretend-synchrony-goolong-2019` - AWS formal methods (2015) - title: Use of Formal Methods at Amazon Web Services (How Amazon Web Services Uses Formal Methods) - what: AWS engineers used TLA+ and TLC to explore distributed designs and find subtle bugs, including in replication and recovery designs. It is the clearest industrial statement of the gap between model and code. - limits: It checks designs only and does not prove the deployed code correct. The linked manuscript is dated 29 September 2014 and differs from the final publication. - how much was read: parts read - source; notes: P01. How Amazon Web Services Uses Formal Methods; notes: papers - record id: `aws-formal-methods-2015` - P language (2013) - title: P: Safe Asynchronous Event-Driven Programming - what: P programs are communicating state machines. Some are executable. Others are ghost machines that exist only for specification. The compiler also generates C code, which was used for a Windows 8 driver stack. - limits: Systematic testing leaves liveness checks for future work (PDF p. 9). Foreign functions and runtime integration need separate treatment. Do not read the historical USB result as saying that current AWS code is generated from P. - how much was read: parts read - source; notes: P06. P: Safe Asynchronous Event-Driven Programming - record id: `p-language-2013` - Property Specification Patterns (1999) - title: Patterns in Property Specifications for Finite-state Verification - what: This is a catalog of reusable requirement patterns, about what occurs and in what order. It also lists scopes: global, before, after, between, and after-until. Each pattern is mapped to formal languages for describing behavior over time. The authors also surveyed real specifications. - limits: The notes read the authors' pattern repository, not the ICSE PDF. The repository labels a formula about all branching futures (AG/AF) as LTL. - how much was read: parts read - source - record id: `property-specification-patterns-1999` - CockroachDB TLA+ models - what: ParallelCommits.tla states progress assertions: the transaction record is eventually aborted or committed, and all intents (pending writes) are eventually resolved. The central property in StoreLiveness.tla, 'Durable Support', is a safety obligation despite its name. - limits: These are models only, with no proof that code matches them. Classify properties by their formula, not by file name. - how much was read: parts read - source; notes: R07. etcd/Raft, TiDB, Elastic, and CockroachDB - record id: `cockroachdb-tla-models` - Elastic formal models - title: elasticsearch-formal-models - what: The repository has paired TLA+ and Isabelle models of Elasticsearch cluster coordination and data replication. The coordination state maps one-to-one onto the implementation. - limits: The models can differ from the implementations a lot. The one-to-one match holds only for coordination state. - how much was read: parts read - source; notes: R07. etcd/Raft, TiDB, Elastic, and CockroachDB - record id: `elastic-formal-models` - etcd Raft TLA - title: etcd/Raft TLA+ trace-validation integration - what: Build with the with_tla tag so that instrumented Go executions produce events, which are checked against the Raft model. - limits: There is no theorem that the whole code matches the model. Ongaro's original Raft model is a separate artifact. - how much was read: parts read - source; notes: R07. etcd/Raft, TiDB, Elastic, and CockroachDB - record id: `etcd-raft-tla` - Ivy liveness examples - what: Official Ivy examples: ticket scheduling, FIFO, cascaded, blocked and bypassing queues, and model refinement. ticket.ivy is the running example of the POPL'18 paper. ticket_nested.ivy needs temporal prophecy (guesses about the future). - limits: These are abstract proof models. The review did not run the checker. - how much was read: parts read - source; notes: additional temporal methods - record id: `ivy-liveness-examples` - TiDB TLA+ models - title: pingcap/tla-plus (TiDB/TiKV models) - what: TLA+ models for Raft, RaftMerge, TwoPC, ConcurrentPercolator, DistributedTransaction and ResolvedTS. TwoPC also has TLAPS material. - limits: There is no theorem linking the models to a common implementation, and no measured effort. Inspect each model's properties. - how much was read: parts read - source; notes: R07. etcd/Raft, TiDB, Elastic, and CockroachDB - record id: `tidb-tla-models` - TLA-Prove - what: This is the public repository of TLA-Prover at commit 6427de9. The audit found two case lists (a 30-row holdout and a 20-row suite called 30-case). It found a saved result (n=30, 1 Diamond, 3 Gold, retrieval on) that differs from the paper. It also found a mutation check that differs from the paper. - number: holdout_v15.json: n=30, diamond=1, diamond_rate=0.033, three Gold - limits: The notes did not reproduce the paper's results, and did not rerun training or inference. The validator graded a two-state toggle with TypeOK as Gold and Diamond. This shows that Diamond does not check whether the spec fits the task. No trained model was run. - how much was read: parts read - source; notes: artifact observations - record id: `tla-prove` - TLAPS liveness examples - title: tlapm examples (SimpleEventually, Dekker, Allocator) - what: These are example TLA+ proofs. SimpleEventually is a 73-line proof that a toy spec eventually sets x to TRUE. Dekker (978 lines) proves mutual exclusion, no-stuck and starvation freedom. Allocator states liveness properties but proves only safety theorems. - number: SimpleEventually: 73 lines, four proof steps - number: Dekker: 978 lines, two steps left OMITTED - number: Allocator: 297 lines, no THEOREM for its three liveness properties - limits: By the notes' inference, there is exactly one fully proved non-toy liveness example, Dekker, with two omitted steps. The other proofs listed (Paxos, Bakery, Peterson, EWD840) are about safety. - how much was read: parts read - source - record id: `tlaps-liveness-examples` - COBALT-TLA (2026) - title: COBALT-TLA: A Neuro-Symbolic Verification Loop for Cross-Chain Bridge Vulnerability Discovery - what: Cobalt pairs an LLM with a TLC REPL. The LLM generates bounded TLA+ models of cross-chain bridges and gets structured traces back so it can repair them. The work was motivated by bugs in the order of events in cross-chain bridges. - number: three bridge targets including Nomad - limits: This is a model-checking workflow. Nothing proves that the generated spec is equivalent to the bridge code. - how much was read: only abstract or first pages read - source - record id: `cobalt-tla-2026` - CONCUR (2026) - what: A benchmark of 43 concurrency problems from a textbook, plus 72 validated mutants (deliberately broken variants). It evaluates LLM-written concurrent code by model checking it for deadlock and race. - number: 115 total problems - limits: It has no proofs. - how much was read: only abstract or first pages read - source - record id: `concur-2026` - Deterministic Execution and Margin Enforcement in Financial Transaction Systems: Empirical Evaluation with Formal Safety Verification (2026) - what: Proposes an order-handling design for trading in which every balance change is appended to a ledger and margin is reserved explicitly, models the accounting core in TLA+, and compares it with two common designs under concurrent load. - number: Median order latency 9.39 ms, p99 33.70 ms - number: Optimistic locking accepted 609,300 of orders against $100,000 of capital (509.3% over-commitment); the append-only ledger had zero false acceptances
  • limits: Only the abstract was available (the TechRxiv PDF page returned an HTML page, not the paper); state-space size, invariant text, model effort and proof size are unknown. The abstract says the property holds ‘within the verified state space’, so it is a bounded model check, not a proof.
  • how much was read: only abstract or first pages read
  • source; notes: ## Deterministic margin enforcement with TLA+ (2026)
  • record id: deterministic-execution-and-margin-2026
• Generalized DBLog (2026)
  • title: Generalized DBLog: A Verified Contract for Interleaving Copied Rows with a Change Log
  • what: This is a verified contract for interleaving table copies with a change log. It covers Netflix DBLog, Debezium and Flink CDC.
  • limits: The conditions are of the safety kind (nothing bad happens).
  • how much was read: only abstract or first pages read
  • source
  • record id: generalized-dblog-2026
• Isolde (2026)
  • what: Isolde is a SAT tool, with no LLM. It produces examples that one isolation level allows and another does not. It also checks whether two definitions are equal.
  • limits: The notes have only a summary.
  • how much was read: only abstract or first pages read
  • source
  • record id: isolde-2026
• VCoT-Bench (2026)
  • what: VCoT-Bench has 1,988 tasks that reconstruct Verus proof steps.
  • number: 1,988 tasks
  • how much was read: only abstract or first pages read
  • source
  • record id: vcot-bench-2026
• Fortifying Integrity, Traceability, and Operational Efficacy in Blockchain-Based Examination Management Through Formal Verification with TLA+ (2025)
  • what: Models an Ethereum smart contract for exam records in TLA+ (a specification language for state machines), checks it with the TLC model checker for safety and liveness, and maps the TLA+ constructs to Solidity code. Hoare logic (pre- and postconditions) is applied to key functions.
  • limits: Full text not obtained (IEEE paywall, no open copy found by Unpaywall or Semantic Scholar); only the abstract was read, so properties checked, state-space size and results are unknown.
  • how much was read: only abstract or first pages read
  • source; notes: TLA+ for a blockchain exam-management contract (2025)
  • record id: fortifying-integrity-traceability-and-2025
• Machine Verification of Correctness for a Concurrent Union-Find Object (2025)
  • what: A machine-checked proof, written in TLA+ and checked by TLAPS, that the Jayanti-Tarjan randomized-linking union-find objects are strongly linearizable (a property that keeps concurrent behaviour correct even with randomized callers).
  • number: proof spans over 16,000 lines
  • number: organized into 20 theorems
  • limits: Full text not obtained: the Dartmouth Digital Commons PDF link returned a bot-check page and I did not work around it. Only the landing-page abstract was read.
  • how much was read: only abstract or first pages read
  • source; notes: ## Karun Ram: union-find in TLA+ (abstract only)
  • record id: machine-verification-of-correctness-2025
• Advances in Automatic Software Verification: SV-COMP 2020
  • what: . This report describes the 2020 Competition on Software Veri- fication (SV-COMP), the 9th edition of a series of comparative evaluations of fully automatic software verifiers for C and Java programs. The compe- tition provides a snapshot of the current state of the art in the area, and has a strong focus on replicability of its results.
  • how much was read: only abstract or first pages read
  • source
  • record id: advances-in-automatic-software
• Automatic Verification of C and Java Programs: SV-COMP 2019
  • what: . This report describes the 2019 Competition on Software Verification (SV-COMP), the 8th edition of a series of comparative evaluations of fully automatic software verifiers for C programs, and now also for Java programs. The competition provides a snapshot of the current state of the art in the area, and has a strong focus on replicability of its results.
  • how much was read: only abstract or first pages read
  • source
  • record id: automatic-verification-of-c
• CASP: An Evaluation Dataset for Formal Verification of C Code
  • what: . Recent developments in Large Language Models (LLMs) have shown promise in automating code generation, yet the generated programs lack rigorous correctness guarantees. Formal verification can address this shortcoming, but requires expertise and is time-consuming to apply.
  • how much was read: only abstract or first pages read
  • source
  • record id: casp-an-evaluation-dataset
• Comparative Analysis of Hoare Logic-Based Formal Verification Tools for Solidity Smart Contracts
  • what: Formal verification of smart contracts is widely regarded as an effective method for ensuring correctness and security properties across all possible executions. Its practical relevance has been driven by the availability of automatic verification tools that discharge intricate proofs. Another area of growing interest is the integration of specification paradigms — for example, combining Hoare-logic–style specifications (pre/postconditions and invari- ants) with SMT and symbolic reasoning — so that each technique can precisely capture complementary aspects of contract behavior.
  • how much was read: only abstract or first pages read
  • source
  • record id: comparative-analysis-of-hoare
• Competition on Software Verification and Witness Validation: SV-COMP 2023.
  • what: . The 12th edition of the Competition on Software Verification (SV-COMP 2023) is again the largest overview of tools for software verifi- cation, evaluating 52 verification systems from 34 teams from 10 countries. Besides providing an overview of the state of the art in automatic software verification, the goal of the competition is to establish standards, provide a platform for exchange to developers of such tools, educate PhD students on reproducibility approaches and benchmarking, and provide comput- ing resources to developers that do not have access to compute clusters.
  • how much was read: only abstract or first pages read
  • source
  • record id: competition-on-software-verification
• Deductive Verification of Unmodified Linux Kernel Library Functions
  • what: . This paper presents results from the development and evalu- ation of a deductive verification benchmark consisting of 26 unmodified Linux kernel library functions implementing conventional memory and string operations. The formal contract of the functions was extracted from their source code and was represented in the form of preconditions and postconditions.
  • how much was read: only abstract or first pages read
  • source
  • record id: deductive-verification-of-unmodified
• Formally Modelling the Rijkswaterstaat Tunnel Control Systems in a Constrained Industrial Environment
  • what: Formally Modelling the Rijkswaterstaat Tunnel Control Systems in a Constrained Industrial Environment Kevin H.J. Jilissen Peter Dieleman Jan Friso Groote Rijkswaterstaat Rijkswaterstaat Eindhoven University of Technology Utrecht, the Netherlands Utrecht, the Netherlands Eindhoven, the Netherlands Eindhoven University of Technology peter.dieleman@rws.nl j.f.groote@tue.nl Eindhoven, the Netherlands kevin.jilissen@rws.nl Rijkswaterstaat, the National Dutch body responsible for infrastructure, recognised the importance of formal modelling and set up a program to model the control of road tunnels. This is done to improve the standardisation of tunnel control and make communication with suppliers smoother.
  • how much was read: only abstract or first pages read
  • source
  • record id: formally-modelling-the-rijkswaterstaat
• FVEval: Understanding Language Model Capabilities in Formal Verification of Digital Hardware
  • what: The remarkable reasoning and code generation capabilities of large language models (LLMs) have spurred significant interest in applying LLMs to enable task automation in digital chip design. In particular, recent work has investigated early ideas of applying these models to formal verification (FV), an approach to verifying hardware implementations that can provide strong guarantees of confidence but demands significant amounts of human effort. While the value of LLM-driven automation is evident, our understanding of model performance, however, has been hindered by the lack of holistic evaluation.
  • how much was read: only abstract or first pages read
  • source
  • record id: fveval-understanding-language-model
• Improvements in Software Verification and Witness Validation: SV-COMP 2025.
  • what: . The 14th edition of the Competition on Software Verification (SV-COMP 2025) evaluated 62 verification tools and 18 witness validation tools, making it the largest comparison of its kind so far. Out of these, 35 verification and 13 validation tools participated with an active sup- port of teams led by 33 different representatives from 12 countries.
  • how much was read: only abstract or first pages read
  • source
  • record id: improvements-in-software-verification
• Mining the Archive of Formal Proofs.
  • what: . The Archive of Formal Proofs is a vast collection of computer-checked proofs developed using the proof assistant Isabelle. We perform an in-depth anal- ysis of the archive, looking at various properties of the proof developments, in- cluding size, dependencies, and proof style.
  • how much was read: only abstract or first pages read
  • source
  • record id: mining-the-archive-of
• Progress on Software Verification: SV-COMP 2022.
  • what: . The 11th edition of the Competition on Software Verification (SV-COMP 2022) provides the largest ever overview of tools for software verification. The competition is an annual comparative evaluation of fully automatic software verifiers for C and Java programs.
  • how much was read: only abstract or first pages read
  • source
  • record id: progress-on-software-verification
• RERS 2019: Combining Synthesis with Real-World Models.
  • what: . This paper covers the Rigorous Examination of Reactive Sys- tems (RERS) Challenge 2019. For the first time in the history of RERS, the challenge features industrial tracks where benchmark programs that participants need to analyze are synthesized from real-world models.
  • how much was read: only abstract or first pages read
  • source
  • record id: rers-2019-combining-synthesis
• Reusable Specification Patterns for Verification of Resilience in Autonomous Hybrid Systems
  • what: . Autonomous hybrid systems are systems that combine dis- crete and continuous behavior with autonomous decision-making, e.g., using reinforcement learning. Such systems are increasingly used in safety-critical applications such as self-driving cars, autonomous robots or water supply systems.
  • how much was read: only abstract or first pages read
  • source
  • record id: reusable-specification-patterns-for
• Software Verification: 10th Comparative Evaluation (SV-COMP 2021).
  • what: . SV-COMP 2021 is the 10th edition of the Competition on Software Verification (SV-COMP), which is an annual comparative eval- uation of fully automatic software verifiers for C and Java programs. The competition provides a snapshot of the current state of the art in the area, and has a strong focus on reproducibility of its results.
  • how much was read: only abstract or first pages read
  • source
  • record id: software-verification-10th-comparative
• Software Verification and Verifiable Witnesses - (Report on SV-COMP 2015).
  • what: . SV-COMP 2015 marks the start of a new epoch of software verification: In the 4th Competition on Software Verification, software ver- ifiers produced for each reported property violation a machine-readable error witness in a common exchange format (so far restricted to reacha- bility properties of sequential programs without recursion). Error paths were reported previously, but always in different, incompatible formats, often insufficient to reproduce the identified bug, and thus, useless to the user.
  • how much was read: only abstract or first pages read
  • source
  • record id: software-verification-and-verifiable
• Status Report on Software Verification - (Competition Summary SV-COMP 2014).
  • what: . This report describes the 3rd International Competition on Software Verification (SV-COMP 2014), which is the third edition of a thorough comparative evaluation of fully automatic software verifiers. The reported results represent the state of the art in automatic soft- ware verification, in terms of effectiveness and efficiency.
  • how much was read: only abstract or first pages read
  • source
  • record id: status-report-on-software
• Syntroids: Synthesizing a Game for FPGAs using Temporal Logic Specifications.
  • what: —We present Syntroids, a case study for In this paper, we describe a case study in which we apply the automatic synthesis of hardware from a temporal TSL-based synthesis to the development of a non-trivial arXiv:2101.07232v1 [cs.LO] 18 Jan 2021 logic specification. Syntroids is a space shooter arcade arcade game. We develop Syntroids, a space shooter game game realized on an FPGA, where the control flow architecture has been completely specified in Temporal realized on an FPGA.
  • how much was read: only abstract or first pages read
  • source
  • record id: syntroids-synthesizing-a-game
• Systematic Generation of Diverse Benchmarks for DNN Verification.
  • what: . The field of verification has advanced due to the interplay of theoretical development and empirical evaluation. Benchmarks play an important role in this by supporting the assessment of the state-of-the- art and comparison of alternative verification approaches.
  • how much was read: only abstract or first pages read
  • source
  • record id: systematic-generation-of-diverse
• The 2019 Comparison of Tools for the Analysis of Quantitative Formal Models - (QComp 2019 Competition Report).
  • what: . Quantitative formal models capture probabilistic behaviour, real-time aspects, or general continuous dynamics. A number of tools support their automatic analysis with respect to dependability or perfor- mance properties.
  • how much was read: only abstract or first pages read
  • source
  • record id: the-2019-comparison-of
• The Karlsruhe Java Verification Suite
  • what: . Thanks to the deductive verifier KeY, the formal verification of Java programs has a long-standing tradition at Karlsruhe. The design of KeY implies some properties that can restrict its use in real-world application cases: (1) Verifying long, code-intensive methods with many instructions or bit-wise operations is difficult even if their behaviour is not overly complex, and (2) tracking formal guarantees through unverified code is difficult if not impossible using KeY.
  • how much was read: only abstract or first pages read
  • source
  • record id: the-karlsruhe-java-verification
• The Quantitative Verification Benchmark Set.
  • what: . We present an extensive collection of quantitative models to facilitate the development, comparison, and benchmarking of new verifi- cation algorithms and tools. All models have a formal semantics in terms of extensions of Markov chains, are provided in the Jani format, and are documented by a comprehensive set of metadata.
  • how much was read: only abstract or first pages read
  • source
  • record id: the-quantitative-verification-benchmark
• Tools at the Frontiers of Quantitative Verification - QComp 2023 Competition Report.
  • what: . The analysis of formal models that include quantitative as- pects such as timing or probabilistic choices is performed by quantitative verification tools. Broad and mature tool support is available for com- puting basic properties such as expected rewards on basic models such as Markov chains.
  • how much was read: only abstract or first pages read
  • source
  • record id: tools-at-the-frontiers
• VerifyThis 2015
  • what: VerifyThis 2015 was a one-day program verifi- series after the competitions held at FoVeOOS 2011, FM2012 cation competition which took place on April 12th, 2015 in and Dagstuhl (Seminar 14171, April 2014). London, UK, as part of the European Joint Conferences on The aims of the competition were: Theory and Practice of Software (ETAPS 2015). It was the fourth instalment in the VerifyThis competition series.
  • how much was read: only abstract or first pages read
  • source
  • record id: verifythis-2015
• VerifyThis 2019: a program verification competition
  • what: VerifyThis is a series of program verification competitions that emphasize the human aspect: participants tackle the verification of detailed behavioral properties—something that lies beyond the capabilities of fully automatic verification and requires instead human expertise to suitably encode programs, specifications, and invariants. This paper describes the 8th edition of VerifyThis, which took place at ETAPS 2019 in Prague. Thirteen teams entered the competition, which consisted of three verification challenges and spanned 2 days of work.
  • how much was read: only abstract or first pages read
  • source
  • record id: verifythis-2019-a-program
• VerifyThis – Verification Competition with a Human Factor
  • what: . VerifyThis is a series of competitions that aims to evaluate the current state of deductive tools to prove functional correctness of programs. Such proofs typically require human creativity, and hence it is not possible to measure the performance of tools independently of the skills of its user.
  • how much was read: only abstract or first pages read
  • source
  • record id: verifythis-verification-competition-with
• primal-dual Houdini (2022)
  • title: Induction Duality: Primal-Dual Search for Invariants
  • what: The paper pairs Houdini with a dual search over states. Houdini drops candidate invariants until the remaining ones hold. The method also finds counterexamples for broken protocols.
  • number: Table 2: 20 benchmarks (10 safe, 10 unsafe)
  • number: Fixed VM artifact about 5.45 GB
  • number: Pages 1-29
  • limits: The publisher PDF returned HTTP 403, so the review notes read the artifact instructions and source instead.
  • how much was read: reading depth not recorded
  • source; notes: Primal-dual Houdini: full fixed 20-task evaluation
  • record id: primal-dual-houdini-2022
• FOL-IC3 (2020)
  • title: First-Order Quantified Separators
  • what: Searches for formulas that separate good finite states from bad ones, inside an IC3 proof search. IC3 builds a proof by ruling out states that could lead to a violation.
  • number: Pinned examples/fol directory has 19 files (a manifest, not a count of successes)
  • number: Pages 703-717
  • limits: The list of 19 files is the repository’s contents. It does not claim all of them were proved.
  • how much was read: reading depth not recorded
  • source; notes: FOL-IC3 and mypyvy; notes: FOL-IC3 pinned properties
  • record id: fol-ic3-2020
• Finding bugs in Raft implementations (Antithesis) (2026)
  • what: Reports bugs in every Raft implementation tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, MicroRaft.
  • limits: Known only from a search summary. Not read.
  • how much was read: not read; title and metadata only
  • source
  • record id: finding-bugs-in-raft-2026
• VeriBench (2026)
  • title: VeriBench: An End-to-End Formal Verification Benchmark for AI Coding Agents
  • what: A launch-draft blog for paired Python/Lean verification tasks, scored against human reference theorem statements. The requested paper titles (VeriBench, Veribench-FTP) could not be found as papers.
  • number: gold theorem sorry rate about 0.57 despite claim that all close without sorry
  • limits: There is no paper or artifact. The repositories returned 404. Treat it as a lead.
  • how much was read: not read; title and metadata only
  • source; notes: VeriBench and Veribench-FTP: exact requested publication not recovered
  • record id: veribench-2026
• AWS correctness practices (2025)
  • title: Systems Correctness Practices at Amazon Web Services
  • what: The paper was not read. Search summaries say it covers P, model checking, fuzzing, property-based testing, fault injection and simulation, with formal specs used as test oracles (the source of expected answers).
  • limits: Fetching the paper returned HTTP 403. The claims are unverified search summaries.
  • how much was read: not read; title and metadata only
  • source; notes: P02. Systems Correctness Practices at Amazon Web Services
  • record id: aws-correctness-practices-2025
• Blockchain verification monograph (2025)
  • title: Security analysis and formal verification on blockchain and its applications
  • what: Per its abstract, a monograph that sorts real blockchain correctness and security flaws, and surveys the models, specifications and techniques used to formally verify blockchains.
  • limits: Full text not obtained: publisher PDF is behind a bot-check page (not worked around), no open copy found. Only the publisher abstract from Crossref was read, so nothing about its content beyond the abstract is recorded.
  • how much was read: not read; title and metadata only
  • source; notes: Security analysis and formal verification on blockchain and its applic
  • record id: blockchain-verification-monograph-2025
• Oracle TLA+ talk (2024)
  • title: Reverse-Engineering with TLA+ at Oracle
  • what: The talk covers reverse-engineering TLA+ models at Oracle. The TLA+ Foundation summary says that hundreds of TLA+ specs cover dozens of cloud services.
  • limits: The count comes from the Foundation summary. The review notes did not check it against the video. It is not a proof about Oracle’s codebase. The review notes did not analyze a transcript.
  • how much was read: not read; title and metadata only
  • source; notes: R08. Oracle and Intel: scope of the public evidence
  • record id: oracle-tla-talk-2024
• TaxDC (2016)
  • title: TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems
  • what: A study of 104 real timing-dependent bugs in Cassandra, Hadoop MapReduce, HBase and ZooKeeper, each labelled by what triggers it, how it shows up, and how it was fixed. Full text not obtained; this is from the input file and a web search summary, not from reading the paper.
  • number: 104 bugs in 4 systems (input file and search summary)
  • number: 2,083 classification labels (search summary, not checked in the paper)
  • limits: Full text blocked: ACM page returned HTTP 403, no open copy found on author pages (uchicago.edu host did not respond), Semantic Scholar lists it as closed access. No claims here are from the paper body.
  • how much was read: not read; title and metadata only
  • source; notes: ## TaxDC (2016)
  • record id: taxdc-2016
• Intel pre-RTL (2008)
  • title: Pre-RTL Formal Verification: An Intel Experience
  • what: A historical lead on hardware design. Lamport’s page on industrial use says TLA+ was still used at Intel in 2014.
  • limits: Historical hardware evidence only. It cannot support a 2026 claim. It is not counted among the 24 catalog papers.
  • how much was read: not read; title and metadata only
  • notes: R08. Oracle and Intel: scope of the public evidence
  • record id: intel-pre-rtl-2008
• Auth Invariant Tester: Temporal Authorization Correctness Verification for Distributed Systems
  • what: Per the input list, a tester for whether distributed authorization (caching, delayed revocation, token lifetime) keeps temporal correctness properties; not read.
  • limits: Full text could not be fetched: the Authorea page and DOI link return a Cloudflare ‘Just a moment’ challenge, which I did not work around. Everything here comes from the one-line description in the input file.
  • how much was read: not read; title and metadata only
  • source; notes: ## Auth Invariant Tester (metadata only)
  • record id: auth-invariant-tester-temporal

Last edited: 2026-10-31