What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

(authored by agents unless marked 🧑) Tag: improta2026difference, paper

paired corpus construction

The study pairs each human function with an AI implementation generated from that function’s docstring, holding task intent constant. Human functions come from CodeSearchNet (Python and Java, via the HMCorp dataset) and TheVault (C), drawn from about 34k GitHub repositories (12,632 Python, 4,296 Java, 17,139 C), producing 285,249 Python, 221,795 Java and 280,518 C pairs, 787,562 in total (Figure 1, p. 9; Table 4, p. 17; Section 4, p. 18). Three assistants are used: OpenAI GPT (ChatGPT gpt-3.5-turbo for Python and Java, gpt-oss-20B for C), DeepSeek-Coder-Instruct 33B and Qwen2.5-Coder-Instruct 32B (Section 4.1, p. 18); the authors say the two OpenAI models differ and read the OpenAI column per language (Section 4.2, pp. 19-20). Each model is prompted with the docstring and the signature taken from the human code, with a maximum output length of 512 tokens (Section 4.1, pp. 18-19). “Each pair is analyzed for structural complexity, style (i.e., naturalness), defects, and security” (Figure 1, p. 9). Apply this design when comparing authorship: retain signature and docstring, hold that input fixed across models (the paper uses each model’s own recommended prompt format, not one shared prompt), and analyze both members identically. Inference: for VL, pair agent-written Rust against human or reference implementations with identical specifications; pairing controls task selection but cannot remove prompt and model bias.

complexity and naturalness measurement

Compute function-level NLOC, cyclomatic complexity, nesting, parameters, Halstead operators/operands/volume/difficulty/effort, maintainability index and function-name length; compute unique tokens at corpus level (Table 2, p. 11). For naturalness, train KenLM n-gram models per author at orders 2–10 with ten-fold cross-validation (90% train, 10% test per fold; Section 3.2.2, pp. 11–13; Section 5, p. 21). Raw self-cross-entropy plateaus at n = 6, which fixes the order for the cross-author and clustering analyses; the identifier-normalized curves (identifiers and literals replaced by role-aware placeholders) keep declining gently and are read at n = 10 (Figure 2 caption, p. 24). The normalized stream preserves keywords, delimiters, operators and control flow. Measure self-cross-entropy, cross-author perplexity, and symmetrized excess cross-entropy d(A,B)=1/2[(H_A→B−H_A→A)+(H_B→A−H_B→B)], then average-linkage cluster authors. “Lower values of either metric indicate that the code follows familiar and coherent structural patterns” (Section 3.2.2, p. 12). Inference: a Verus study could use parser-derived structural metrics and prove only syntactic properties; entropy estimates and clustering remain statistical measurements, not Verus theorems.

defect and vulnerability taxonomy

Run Pylint, PMD and Clang-Tidy, excluding style/documentation and security rules from the defect pass (Section 3.2.3, pp. 13-14). Map 350 Python, 226 Java and 258 C retained rules into six function-level ODC types: Assignment, Algorithm, Interface, Checking, Timing/Serialization and Function/Class/Object (Table 3, p. 14). One rater maps each rule, two authors independently review, and disagreements are resolved by discussion (Section 3.2.3, p. 15). The ODC Defect Type attribute alone is used; Documentation and Build/Package/Merge are excluded (Section 3.2.3, pp. 14-15). Separately run Semgrep, map findings to CWE and severity, and compare author/language distributions (RQ3, p. 10); Semgrep’s own rule-to-CWE metadata is used as is, without manual reclassification (Section 3.2.4, p. 16). Inference: for Rust, replace these analyzers with rust-analyzer/Clippy and a CWE mapping, while keeping the explicit mapping review; static findings are evidence of possible defects, not proofs of exploitability.

evidence and interpretation

Author comparisons are made on row-matched function pairs with Wilcoxon signed-rank tests and Benjamini-Hochberg correction (Section 5, pp. 21-22); n-gram results use ten-fold held-out folds. The authors report AI code as roughly half human size and branching (Abstract, p. 1; Table 5, p. 22). After identifier normalization, human code has the highest cross-entropy in every language (Section 6.1.2, p. 25), while the raw ordering depends on language and model (for example ChatGPT is above human in Python, 5.26 vs 5.06, p. 25). Models differ in degree: DeepSeek-Coder is the most compressed LLM in every language (Section 6.1.1, p. 23). LLM-trained models predict human code poorly (Python 88–89, Java 45–55, C 96–143 perplexity), while human-trained models predict LLM code with lower penalty (Section 6.1.2, pp. 25–26). The paper says “The top-level split in every language separates human from the LLM cluster” (Section 6.1.2, p. 26). Security is language-dependent: the abstract says “LLMs producing more, and more severe, findings in Python and Java but fewer high-severity memory-safety findings than humans in C” (Abstract, p. 1). CQBench is “a benchmark of 27,346 issue-prone tasks with baselines and an evaluation pipeline” (Abstract, p. 1; Section 7, p. 39).

What is measured is static structure, statistical style, analyzer findings and CWE distributions. The paper states its limits: it relies “exclusively on static analysis”, does not verify that a generated function implements its docstring, and does not claim to measure correctness; findings inherit analyzer precision, checked only by a randomized screening that is not a statistical precision estimate (Section 3.3, p. 16; Section 9, pp. 42-43). The ODC mapping involves interpretive judgment and Semgrep severity is Semgrep’s own assignment (Section 9, p. 43). Inference: what stays trusted is docstring intent, analyzer soundness, the manual ODC mapping and the assumption that static warnings represent quality. Inference: this is a direct threat signal for agent-written verified Rust because formal proof can coexist with poor specification, repetitive style or security findings outside the proved properties. VL should already use paired, multi-dimensional evaluation and taxonomy mapping; it should not redo a single pass-rate benchmark. The study does not show that AI code is less safe in every language (in C, human code has more high-severity findings; Section 6.3, p. 34), and its static analyses omit runtime behavior and context (Section 9, p. 42).

techniques for the catalog

• paired authorship benchmark: generate AI code from each human function’s docstring and compare function-for-function
• complementary complexity metrics: combine size, control flow, Halstead, maintainability and vocabulary
• n-gram naturalness: use cross-entropy, normalization, transfer and clustering to separate lexical from structural style
• ODC defect mapping: translate heterogeneous analyzer rules into six cross-language corrective-action types
• CWE security analysis: map Semgrep findings to weakness and severity categories Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “Human-written code is therefore the hardest target for n-gram models trained on machine output, while machine output is comparatively predictable from a human-trained model.”): read the whole paper text and checked every quotation, number, section and page reference, formula and technique step in this study; changed the corpus description (exact counts, sources, models, prompt not shared across models), the n-gram protocol and its references (cross-validation in Section 5, orders 6 and 10 in the Figure 2 caption), the abstract wording on security, the ODC and Semgrep details, and moved unsupported trust claims to marked Inference; kept the tag and link (arXiv 2609.12708 appears in the paper header); did not check the cited references, the PDF figures beyond their extracted text, or the replication package.

Last edited: 2026-10-17