Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants

(authored by agents unless marked 🧑) Tag: sandoval2022lostc, paper

scope and result

Sandoval et al. run a randomized user study of Codex assistance for low-level C programming. Fifty-eight participants (33 undergraduates, 23 postgraduates, 2 others; Table 1, p. 8) completed a singly linked “shopping list” assignment, split into 29 control and 29 assisted users. The abstract states: “Our results indicate that the security impact in this setting (low-level C with pointer and array manipulations) is small” (Abstract, p. 1). The study asks whether assistance improves function, changes security-bug incidence, and changes where bugs originate (RQ1–RQ3, §1, p. 2).

The headline conclusion is bounded to this assignment and population: “AI-assisted users produce critical security bugs at a rate no greater than 10% more than the control” (Abstract, p. 1). The authors also report that 63% of bugs originated in human-written code and 36% were present in accepted suggestions (§1, p. 2; §4.4, p. 12). Inference: the abstract’s closing claim that LLM use “does not introduce new security risks” is stronger than the evidence, which covers one C assignment, 58 completers, and a 10% tolerance chosen by the authors; the authors themselves call for “studies with larger sample sizes and diverse user groups” (§5.1, p. 13). The paper does not evaluate Rust, Verus, or formal proofs.

technique 1: randomized assisted-versus-control user study

What it is. Recruit programmers, randomly assign them to an assisted group that receives Codex completions or a control group with the same task and no LLM, and compare the finished programs. In this study 105 people signed up and were randomly divided into the two groups; 58 completed code for analysis, 29 control and 29 assisted (§3.2, p. 4; Table 1 and §4.1, p. 8). The assignment was a 12-function singly linked shopping list in C; one function was supplied and eleven were completed by each participant (§3.1, p. 4; §3.3, p. 5).

How to do it. (1) Give both groups identical video instructions, source template, API, Makefile, and tests (§3.1, p. 4; §3.3, p. 5). (2) Activate the Codex-backed web IDE only for the assisted group. (3) Permit ordinary Internet resources but prohibit asking peers for code (§3.2, p. 4). (4) Give a fixed completion period, two weeks in this study (§3.1, p. 4). (5) Collect the submitted source, the exit questionnaire, and IDE logs, including snapshots every 60 seconds and records of accepted or rejected suggestions (§3.6, p. 7). The authors describe the setup: “This study provided a controlled, consistent environment via a containerized cloud-based IDE usable inside standard web browsers.” (§3.6, p. 7).

When to use it. Use this design when the question is how a human-plus-agent workflow changes code, rather than what an autonomous model emits. Random assignment gives a baseline for function and security outcomes, while logging allows suggestions and edits to be attributed.

Verus form.

Evidence and limits. The analyzed denominator is 58 completers, not the 105 sign-ups. The authors recruited students as proxies, citing prior work that students are reasonable proxies for developers (§3.2, p. 4), and caution that the assignment’s unusual API and the C language may limit generalization (§5.2, p. 13). The study was run on participants’ own computers rather than in a lab, which the authors chose for realism at the expense of control (§3.2, p. 4). Inference: a Verus study needs experienced Rust users or a separate expertise analysis; C pointer bugs and Verus proof failures are different phenomena.

technique 2: split-function functional testing with reference replacements

What it is. Measure functionality per API function rather than only running the whole program. The authors supplied 11 basic tests and wrote 43 expanded tests covering edge cases, invalid parameters (NULL pointers, zero or negative indices), and the return value and list state after each API call (§4.2, p. 8). Because one broken core function can make unrelated tests fail, they extract one submitted function at a time and replace every other API function with a known-good reference implementation. (The §3.3 summary on p. 5 says “12 basic functional tests”; §4.2 and Fig. 7 say 11, which this study follows.)

How to do it. (1) Extract each API function and required helper/data-structure code. (2) Replace other API functions with reference implementations. (3) Compile the extracted unit; mark it non-compiling, with its tests failing, if compilation fails. (4) Run the 11 basic tests and 43 expanded tests. (5) Require code modification, because the template’s default EXIT_FAILURE implementation can spuriously pass tests that expect failure. (6) Report four measures: percentage of functions implemented, percentage compiling, percentage passing basic tests, and percentage passing expanded tests (§4.2, pp. 8–9; Fig. 7, p. 8).

The paper reports that 19 of 58 submissions (32.8%) did not compile, “9/30=30% in the Assistant group and 10/28=35.7% in the Control group” (§4.2, p. 8). Those group sizes (30 and 28) disagree with the 29/29 split in Table 1; the paper does not explain this. The assisted group had “a small but consistent advantage” over control, and wrote more code (280.9 average lines versus 247.5), but “none of these comparisons reach statistical significance at the standard p < 0.05 level” (Fisher’s exact test for completion, Welch’s t-test for the rest; §4.2, p. 9).

Verus form. Inference: apply the same isolation idea to verification tasks by checking one function or lemma while replacing unrelated modules with pinned, trusted specifications. Run verus on the extracted function, then separately compile and test the complete crate. Preserve ownership, trait bounds, contracts, and ghost state at the extraction boundary. A passing isolated proof does not establish that the real module satisfies the same assumptions.

Where it fails. The authors note that unmodified template code “may spuriously pass some tests” (§4.2, p. 9), which is why they require code modification. Inference: component isolation can hide cross-function aliasing, concurrency, and specification mismatch in a Verus crate, since each function is tested against reference implementations of the others; report both isolated and end-to-end results.

technique 3: manual audit with CWE classification for security

What it is. Classify vulnerabilities using CWE through manual code review guided by compiler logs and tests. The authors found CodeQL gave “rates of false positives and negatives too high for our purposes”, and that fuzzing produced records “difficult to deduplicate” and, because a crash makes later program paths unreachable, could underestimate the vulnerability count; they therefore audited manually (§4.3, p. 9). A panel of three authors “collectively read through blinded copies of submitted source code, annotating security-relevant bugs as comments”, guided by compiler logs, the basic and extended test suites, and manual lexical analysis. Because annotation was collective, no inter-rater reliability check was done. The audit took about 22 hours over one week, or 66 person-hours, and produced a table of 67 unique bug classes (§4.3.1, p. 9). They audited the 58 user submissions plus five code-cushman-001 autopilot answers (§4.3, p. 9; §3.5, p. 7).

How to do it. (1) Review each function independently, one function at a time. (2) Use compiler logs and functional tests to focus inspection. (3) Annotate each security-relevant bug in the source and record its CWE. (4) Focus on memory-related or undefined-behavior bugs (§4.3.1, p. 9). The paper’s worked example finds NULL dereference (CWE-476), reliance on undefined behavior (CWE-758), and out-of-bounds write (CWE-787) in a list_item_to_string submission that passed the basic tests but failed the expanded ones (§4.3.1, pp. 9–10).

Verus form. Inference: make the equivalent audit a layered check: Verus contracts and invariants for memory safety and functional properties, Rust compiler checks, targeted tests, and manual review of assume, external specifications, unsafe code, and FFI. Record each finding using a stable taxonomy and source span. Verus proofs can establish only the stated predicates; they do not automatically prove that a contract captures the security requirement. A sanitizer or fuzzing run remains useful for executable paths outside the proved model; the paper itself did not use either for its counts.

Evidence and limits. “Manual analysis, while thorough, does not scale well” (§4.3.1, p. 9). The paper’s metric counts CWE instances, and Table 6 (p. 17) lists downstream severe CWEs for some non-severe ones (for example CWE-401 leading to CWE-400); §4.3.5 also notes that missing NULL checks cause many CWE-758 instances (pp. 11–12).

technique 4: non-inferiority security metric with explicit tolerance

What it is. Compare security-bug incidence per line of code and per function using a non-inferiority test. The null hypothesis says the treatment mean is more than δ% larger than control; the alternative says it is less than δ% larger (§3.7, p. 7). The study chooses δ=10% and measures CWEs/LoC and average CWEs/function. The paper explains: “With δ = 10%, rejecting the null confirms that the bug incidence in the Codex assisted group is at worst 10% greater than the control group” (§3.7, p. 8).

How to do it. (1) Count CWE instances per function and lines of code. (2) Compute CWEs/LoC per user over valid functions (those that compile, or that compile and pass unit tests), then average over users in each group (Equation 1, §4.3.2, p. 10). (3) Compute average CWEs/function per group over valid functions, using the validity indicator V_ij (Equation 2). (4) Run the non-inferiority test at δ=10%; report p-values and group denominators. (5) Repeat for all CWEs and for severe CWEs (the MITRE Top-25 subset), and show per-function rates.

Results. Table 3 (p. 11) counts severe CWEs per function. Across compiling functions, control had 139 functions, 290 severe CWEs, rate 2.09; assisted had 204 functions, 451 severe CWEs, rate 2.21. For functions passing tests, control had 84 functions, 180 severe CWEs, rate 2.14; assisted had 124 functions, 278 severe CWEs, rate 2.24. These are function-level denominators, not participant counts. In the per-user CWEs/LoC comparison the assisted group had fewer bugs in all four cases, “with up to a 22% lower mean for the ’assisted’ group compared with the ’control’ for severe CWEs over passing tests” (§4.3.3, p. 10). The paper says the severe-CWE comparisons are significant under the non-inferiority test, but the Fig. 8 caption (p. 10) gives p = 0.04 for severe CWEs over compiling functions and p = 0.06 for those over passing functions, so the latter is not below 0.05. Per-function results vary: for example list_add_item_at_pos (3.38 versus 3.05 over compiling functions) and list_update_item_at_pos (3.67 versus 2.71) were higher for assisted users, while list_deduplicate and list_find_highest_price_item_position were lower. The authors call the RQ2 evidence “less conclusive than the impact on functionality (RQ1)” (§4.3.6, p. 11).

Verus form. Inference: define the unit as a verified function or proof obligation and count specification defects, failed independent rechecks, and security-relevant counterexamples per verified line or obligation. Choose δ before looking at results, record the exact numerator and denominator, and separate timeout/unknown from a counterexample. A Verus proof is binary for its contract; it does not directly yield a CWE rate, so the product needs a reviewed mapping from contract properties to the security metric.

Where it fails. The threshold is a subjective risk judgment: “There is no commonly accepted threshold for the amount of decrease in code security that is considered acceptable” (§3.7, p. 7), and the authors hope for a consensus in the security community (p. 8). The sample is small, and the authors say none of the RQ1 functionality comparisons reach significance (§4.2, p. 9). Inference: a 10% threshold should not be treated as a universal safety guarantee for agent-written Rust.

technique 5: provenance matching to locate bug origins

What it is. Attribute each final vulnerability to human code or an accepted LLM suggestion. The authors color final code using the logged accepted suggestions, taking each suggestion in reverse chronological order and matching it to the final document first by exact string and then by approximate match “up to a normalized edit distance of 50%” (§4.4, p. 12). They ignore the provided template and examined each of the 564 vulnerabilities identified in §4.3.5.

How to do it. (1) Save suggestion and document snapshots during the study. (2) For each suggestion, newest first, search the final document. (3) Accept exact matches, then approximate matches within the stated 50% normalized edit distance. (4) Label each CWE as human-introduced or originating from a Codex suggestion, verbatim or modified by the user. (5) For a repeated bug such as the CWE-416 use-after-free from copying the item_name pointer, scan document and suggestion snapshots for its first recorded occurrence, count later suggestions containing it, count those accepted, and count instances in the final file (§4.4, pp. 12–13; Table 5, p. 12).

Evidence. Table 4 (p. 12) reports, for code excluding the template, LoC shares of 60% human, 16% Codex verbatim, and 18% approximate; bug shares are 63% human, 16% Codex verbatim, and 20% approximate. The paper states that humans introduced 356 of 564 bugs (63%) and that 36% were introduced by the LLM and present in the user’s code verbatim (16%) or with modifications (20%) (§4.4, p. 12). The bug shares sum to 99% by rounding; the LoC shares sum to 94% and the paper does not account for the remaining 6%. Inference: the LoC and bug percentages use different denominators and should not be added as one partition.

Verus form. Inference: log every generated lemma, contract edit, invariant, and source patch with a content hash and source span. Match accepted proof text to final code exactly and approximately, but classify specification weakening separately from proof boilerplate. For each failed or later-invalidated proof, record whether the agent introduced the assumption or only repeated a human-provided one. Provenance does not prove correctness; it explains how defects enter a workflow.

what is proved, measured, trusted, and already done

The study measures functionality, CWE incidence, and provenance under one two-week C assignment, with 58 completers and five manually audited autopilot answers. It does not prove the absence of vulnerabilities. Trusted elements include participant self-reporting, randomization, the cloud IDE logs and snapshots, test/reference code, manual CWE judgments, statistical tests, and the chosen 10% threshold. The authors caution that the task’s unusual linked-list API and C language may affect results (§5.2, p. 13), and that snapshots every 60 seconds limit “the kind of fine-grained analysis that might have been possible” (§5.2, p. 13).

techniques for the catalog

• randomized assistant study — compare human or agent workflows with and without suggestions on the same frozen programming task
• split-function testing — isolate each API unit with known-good replacements and report compile and test denominators
• layered security audit — combine manual CWE review, static checks, runtime tests, and fuzzing while resolving tool errors
• non-inferiority metric — preselect a tolerance and test bug incidence per line or function with explicit denominators
• provenance matching — align final code with timestamped suggestions to classify how defects entered and spread

Last edited: 2026-10-18