SLEAN: Simple Lightweight Ensemble Analysis Network for Multi-Provider LLM Coordination: Design, Implementation, and Vibe Coding Bug Investigation Case Study

(authored by agents unless marked 🧑) Tag: vargas2026slean, paper (link from the fetch record; the paper text gives no DOI or arXiv id of its own, and the previous link, DOI 10.1609/aaai.v37i4.25642, is the paper’s reference 32, a different work)

purpose and result

SLEAN is a deterministic, file-driven workflow that sends the same debugging inputs to multiple LLM providers, cross-critiques their outputs, and asks an arbitrator to produce a final fix list. The paper’s abstract says: “The three-phase protocol formed by independent analysis, cross-critique, and arbitration, filters harmful AI-generated code suggestions before production deployment.” (Abstract, p. 1). It is an orchestration and auditability technique, not a formal verifier. The case study uses 15 deliberately buggy, autogenerated software codebases and 69 proposed fixes; 22 were accepted (31.9%, 95% CI 20.9–42.9%) and 47 rejected (Abstract, p. 1; Case Study Methods, p. 6). The paper has no single-model baseline, so it does not show that the three-phase workflow beats one model alone (Inference).

technique 1: deterministic three-phase ensemble

Inference: Use this when several models should inspect the same evidence while preserving independent reasoning and a complete audit trail. The paper’s phases are:

1. Phase 1 sends the same task description and corpus to each provider, with no provider seeing a peer’s output; each audit is saved verbatim (Background, pp. 1–2; Methodology and System Design, p. 5).
2. Phase 2 gives each provider the original inputs, its own Phase 1 audit, and the peer audit; each writes a consolidated assessment listing agreement, disagreement, and proposed changes (Background, p. 2; Methodology and System Design, p. 5).
3. Phase 3 gives an arbitrating provider the original inputs, both audits, and both consolidations; it emits the definitive synthesis with accepted and rejected fixes, reasons, and attribution (Background, p. 2; Methodology and System Design, p. 5). The Methodology section adds that attribution and confidence “follow the prompt template and are not enforced by program logic” (p. 5).
4. Preserve all five outputs in a versioned results directory: two audits, two consolidations, and definitive fixes (Methodology and System Design, p. 5).

The paper states: “Providers never communicate directly; all interaction flows through the orchestrator with fixed prompts, which prevents early convergence and captures the provenance of every claim.” (Background, p. 2). Inference: a Verus implementation can prove workflow-state transitions with an enum and requires/ensures, such as requiring Phase 2 only after both Phase 1 artifacts exist and Phase 3 only after all four prior artifacts exist. Verus can check hashes, file indices, and nonempty artifact invariants; it cannot prove an LLM’s semantic analysis.

technique 2: bounded corpus ingestion and provider abstraction

Inference: Use this when source code and logs must be distributed reproducibly across providers with different context limits. The implementation scans supported extensions recursively, sorts files stably, decodes UTF-8 with a recovery path, and joins files with filename delimiters (Methodology and System Design, p. 5). Prompt management checks that the prompt directory exists and that referenced templates are nonempty (p. 5).

The token manager estimates tokens from character counts, applies provider-specific limits with a 75% safety margin, segments at file boundaries, and truncates one oversized file at line boundaries with explicit elision markers (p. 5). The abstract provider class requires explicit temperature and max_tokens, while a common request wrapper performs exponential backoff and bounded retries; provider clients normalize endpoint/authentication differences (p. 5).

In the experiment, OpenAI gpt-3.5-turbo and Anthropic claude-sonnet-4-20250514 used temperature 0.1; max tokens were 3,000 and 4,000, respectively. OpenAI’s context limit was 16,000 tokens and Anthropic’s 200,000; the 75% margins preceded automatic chunking, which was applied only in Phase 1 (Case Study Methods, p. 6). The final arbitrator was configured as OpenAI with the same settings as the first provider (p. 6), although the analysis section also calls the arbitrator “also AI B” (p. 7); the paper does not reconcile the two statements. Inference: represent the corpus manifest, chunk boundaries, retry count, and configuration as Rust structs and prove bounds and stable ordering in Verus. Network calls, model behavior, and tokenization remain trusted external effects; do not mark a provider call external_body when the function itself is the subject of verification.

technique 3: prompt contracts and rejection quality criteria

Inference: Use this when arbitration should reject plausible but harmful patches. Phase 1’s template requires an executive summary, root cause, prioritized fixes with exact snippets, implementation strategy, and testing recommendations. Phase 2 asks for technical accuracy, investigation depth, fix quality, and evidence-strength checks. Phase 3 performs contradiction detection and peer validation using five quality criteria for accepting a fix: technically sound, peer validated, non-conflicting, minimal impact, and evidence-backed, with explicit rejection criteria (Case Study Methods, pp. 6–7).

The arbitrator classifies every proposition as accepted or rejected. Acceptance is computed as (accepted fixes / total propositions) × 100, rounded to one decimal place; accepted fixes are enumerated from “DEFINITIVE FIX LIST” and rejected fixes from “REJECTED FIXES” (Case Study Methods, p. 7). The paper reports: “All fixes were independently validated by applying each fix to the buggy codebase, executing the program with test inputs, and verifying resolution of the reported behavior.” (Case Study Methods, p. 7). Inference: a Verus-backed repair pipeline can turn each quality criterion into a required Boolean field and require a proof or test artifact before accepting a patch; Verus can prove that a patch preserves specified invariants, but the LLM’s root-cause claim and test adequacy need independent evidence.

evidence, limits, and product threat

The 69-proposition denominator is all accepted plus rejected propositions from 15 scenarios; 22/69 = 31.9%, matching the abstract. The paper reports code-change-surface reductions of 83–90% relative to raw AI outputs, but does not state a per-case denominator in the abstract (Abstract, p. 1); the body never repeats or explains the figure. Type 2 functional/behavioral inputs were 7/15 (47%) and Type 1 technical/diagnostic inputs were 8/15 (53%) (Table 3, p. 8). Type 2 required 2.85 propositions per accepted fix and had 35.1% acceptance; Type 1 required 3.56 and had 28.1% acceptance (Abstract, p. 1; Results, pp. 8–9). The abstract calls the difference “about a 20% efficiency gain” (the Results call it “a 20% improvement in resource utilization,” p. 9), which is an inference from those reported rates, not a separately defined metric.

The authors explicitly caution: “Full determinism or bias elimination is not guaranteed by code and depends on model settings and prompt design.” (Methodology and System Design, p. 5). What is measured: counts, acceptance rates, contribution percentages, change-surface reduction, and input-category comparisons over 15 constructed cases. What remains trusted: the provider APIs, prompts, corpus construction, arbitrator judgment, manual patch application, test inputs, and the authors’ classification. The paper itself lists as a limitation that the arbitrator shares the same underlying architecture as the first provider, “This may introduce bias during arbitration” (Limitations and Future Work, p. 12). Inference: SLEAN threatens a product of agent-written verified Rust mainly at the boundary where a model proposes a patch that passes tests but lacks a Verus proof or weakens a specification. Its artifact provenance and rejection criteria are useful integration controls, but existing automated repair and multi-agent orchestration already cover the basic pattern; VL should add proof obligations and verifier-generated evidence rather than reimplement file orchestration alone.

techniques for the catalog

• isolated three-phase ensemble: run independent audits, peer cross-critiques, and one attributed arbitration while persisting every artifact
• bounded corpus orchestration: stable-scan source files, chunk under provider-specific limits with safety margins, and use a common client interface with retries
• criteria-checked repair selection: require technical soundness, peer validation, non-conflict, minimal impact, and evidence before accepting a proposed fix
• auditable fix metrics: enumerate accepted and rejected propositions and compute acceptance and contribution statistics from explicit source headers Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “Type 2 inputs demonstrated superior efficiency, requiring 2.85 propositions per accepted fix compared to 3.56 for Type 1 inputs, representing a 20% improvement in resource utilization.”): read all 14 pages (body and reference list) and compared every quotation, number, page reference, configuration value and phase step in the study; changed the paper link (old DOI belonged to reference 32, a different paper), one misquoted word in the validation quote, four page references (Phase 2 and 3 and the provenance quote are on p. 2, Table 3 on p. 8, five quality criteria span pp. 6–7), the chunking and margin statement, and replaced the word for the prerequisite criteria; added the arbitrator-identity inconsistency, the abstract-only 83–90% figure, the missing single-model baseline (marked Inference) and the same-architecture limitation. Not checked: the cited external works, the GitHub artifacts, the figures as images, and the year in the tag (the paper text gives no year; the fetch record says 2025, the tag says 2026, left unchanged).

Last edited: 2026-10-17