Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware

(authored by agents unless marked 🧑) Tag: chen2025marin, paper

what the paper does

The paper presents MARIN, a two-phase system for reducing hallucinations in project-specific API calls generated by an LLM. It targets project-specific APIs, where isolated retrieval snippets omit the relationships that make a call valid. The abstract states: “Dependency Constrained Decoding, which utilizes mined dependencies to adaptively constrain the generation process, aiming to ensure the generated APIs align with the project’s specifications.” (abstract, p. 1). The evaluation uses six open-source code models, a 416-sample Java benchmark, and an additional 109-sample Huawei benchmark.

technique: hierarchical dependency mining

what it is

The first phase statically extracts local dependencies (called functions and referenced APIs) and global dependencies (the current and related files). It then puts this structural representation into the model prompt. The motivation example shows a Reaction.willReact() call: retrieval gives a snippet that constructs a Reaction and calls setReact, but omits the willReact declaration in the Reaction class, so unconstrained generation emits token fragments get, Re, act, ()) instead of willReact (section 1, p. 2).

when to use it

Inference: Use it when a completion depends on declarations spread across files or modules, especially project-specific APIs. The preliminary study reports 85.25% average hallucination for project-specific APIs and 12.75% for third-party APIs. The authors attribute the difference to training exposure: “they struggle with project-specific APIs due to a lack of contextual awareness.” (section 2.1, p. 3). Inference: a verifier-oriented code assistant should treat repository declarations and signatures as mandatory context rather than optional retrieved examples.

how to do it

1. Start with the project and an incomplete function (section 3.1, p. 3).
2. Use static analysis to identify valid reference APIs at the generation position and called functions within the same file or other project files (section 3.2.1, p. 4).
3. Extract imports from the current file and identify the imported files. Keep class definitions, field declarations, and function signatures as file skeletons, excluding implementations (section 3.2.2, p. 4).
4. Construct a prompt with the project description, skeletons of related and current files, called functions, reference APIs, and the incomplete function with its [API_Position] marker (section 3.2.3 and Figure 4, p. 4).
5. Supply this prompt to the code model before constrained decoding (section 3.1, p. 3).

Verus form

Inference: The analogous Verus step indexes Rust modules, traits, functions, data types, and existing specifications, then adds the resolved declarations and signatures to the generation context. Rust’s module and trait resolution can provide the dependency graph; Verus contracts and relevant spec fn definitions should be included where they constrain a proof. This is context construction, not a proof: Verus must still compile and verify the generated artifact.

what evidence shows it works

APIHulBench contains 416 unique samples from 98 recent Java repositories: APIHulBench-F has 221 samples with the API call in the first half of the function, and APIHulBench-M has 195 with the call after the halfway point (section 4.2, pp. 5–6). The six models are CodeLlama-7B/13B/34B and DeepSeekCoder-1.3B/6.7B/33B (section 4.3, p. 6). In the ablation, removing global dependencies causes a 33.39% drop in average exact match and a 186% rise in average MaHR on APIHulBench-F, averaged across six models; removing local dependencies causes a 15.79% exact-match decrease and an 81.66% MaHR increase (section 5.2, p. 7). The authors state: “In contrast, removing local dependencies results in a 15.79% decrease in average EM and an 81.66% increase in average MaHR.” (section 5.2, p. 7). The passage does not explicitly state the percentage-change denominator or whether these comparisons use constrained decoding.

what fails or is assumed

Inference: Dependency extraction must identify the relevant declarations, and input truncation can limit which reach the model. The implementation limits the input to 8,000 tokens, with 7,000 for project content and 1,000 for function content, truncating excess content from left to right (section 4.6, p. 6). Inference: truncation can remove the declaration needed to disambiguate a call. The authors state: “Our current implementation and evaluation are centered on Java projects.” (section 6.2, p. 10). Inference: The results do not establish that the extractor handles Rust macros, trait implementations, or Verus specifications.

technique: dependency constrained decoding

what it is

MARIN builds an API-name prefix tree and token masks from the mined declarations. At each decoding step, the current trie node determines valid next API-name tokens. A parameter-pattern indicator separately distinguishes calls with and without arguments. The paper explains: “To ensure the generated APIs conform to mined dependencies, we employ an adaptive masking mechanism that combines both API name and parameter pattern constraints.” (section 3.3.2, p. 5).

when to use it

Inference: Use it when project analysis can enumerate valid API names and map them to model tokens. Its constraints target API names and the distinction between calls with and without parameters.

how to do it

1. Tokenize both each full API call and its prefix before the last dot, then remove the shared prefix tokens to account for context-dependent tokenization. Insert the resulting API-name sequences into a prefix tree, sharing common prefixes (section 3.3.1, p. 5).
2. Build parameter-pattern token sets for no-argument and argument-taking calls (section 3.3.1, p. 5).
3. At decoding step t, inspect the current token sequence and trie node (section 3.3.2, p. 5).
4. If the next token is an API-name token, obtain valid next tokens from the children of the current trie node (section 3.3.2, p. 5).
5. If the next token starts parameters, consult the indicator and allow only the matching parameter-pattern set (section 3.3.2, p. 5).
6. Combine these constraints into a binary mask. The paper specifies element-wise multiplication of logits by the mask, followed by softmax and greedy selection. Repeat until a complete API call is generated or the maximum length is reached (equations 1–2, section 3.3.2, p. 5).

Verus form

Inference: A Verus assistant can construct a prefix trie over in-scope Rust and Verus identifiers, including functions, methods, proof functions, and specification names, then mask only identifier positions. A second checker must reject unresolved names, type errors, illegal proof/runtime use, and invalid contracts. The paper’s mask constrains names and parameter patterns; it does not prove a function’s requires/ensures or establish an invariant.

what evidence shows it works

Table 1 reports, for each of six models and both benchmark variants, exact match (EM), edit similarity (ES), identifier match (IM), MaHR, and MiHN. For example, on the CodeLlama-7B row, MARIN on APIHulBench-F is 76.01 EM, 87.04 ES, 87.33 IM, 15.83 MaHR, and 0.17 MiHN; the corresponding APIHulBench-M row is 66.66, 83.66, 89.23, 12.30, and 0.13 (Table 1, p. 7). These are percentages for the first four metrics and a count-like average for MiHN, over the 221-sample F or 195-sample M benchmark as applicable. The paper reports a Wilcoxon signed-rank test with p-value < 0.001 for improvements (section 5.1, p. 6). Its answer to RQ1 states: “achieving an average increase of 107.32% in EM and an average decrease of 67.31% in MaHR compared to the RAG approach.” (section 5.1, p. 7). Table 1’s caption calls its significance test a t-test, whereas section 5.1 calls it a Wilcoxon signed-rank test; the paper does not reconcile the labels.

what fails or is assumed

Inference: Name and parameter-pattern constraints do not validate argument values or prove semantic correctness. They depend on correct tokenization and dependency extraction. Also, multiplying an invalid token’s logit by zero does not exclude it from softmax; zero can exceed valid negative logits. Equation 2 therefore does not, as written, guarantee the stated exclusion. The implementation was not inspected to resolve this discrepancy. For Verus, identifier filtering would still need compilation and verification to detect specification violations.

technique: measurable hallucination and ablation

The paper defines Micro Hallucination Number (MiHN) as the average number of hallucinated elements inside each generated API, and Macro Hallucination Rate (MaHR) as the fraction of generated APIs containing any hallucinated element (section 4.5.2, p. 6). Its exact definition is: “Macro Hallucination Rate (MaHR) calculates the proportion of generated APIs that contain any hallucinatory elements:” (section 4.5.2, p. 6). The denominator is the total number n of generated APIs. Accuracy uses EM, ES, and IM, where EM is exact output match, ES is normalized Levenshtein similarity, and IM is correct API-identifier percentage (section 4.5.1, p. 6).

To isolate causes, the ablation runs eight variants: all dependencies with or without constrained decoding, no local dependencies, no global dependencies, or neither (section 5.2, p. 7). On APIHulBench-F, adding constrained decoding to complete dependencies raises average EM by 9.12 points and lowers MaHR by 11.53 points across six models; on APIHulBench-M it raises EM by 11.28 points and lowers MaHR by 12.22 points (section 5.2, p. 7). The paper concludes: “Among these, global dependency is the most important module, having the greatest impact on overall performance.” (continuation of the answer to RQ2, section 5.2, p. 9).

Verus form

Inference: The same measurement can count unresolved identifiers, compiler errors, failed Verus obligations, and contract-relevant API misuse separately. An exact-match score is less informative for proof code than verification success, so a Verus evaluation should report the denominator of generated functions and the number that compile, verify, and satisfy independent tests.

what is proved, measured, and trusted

Section 4.5 (p. 6) evaluates accuracy using EM, ES, and IM and evaluates hallucination using API names and parameter patterns. Inference: These measurements do not prove functional correctness. The reported validity depends on extracted declarations, tokenization, constraint implementation, and benchmark labels. A Verus product would need separate verification and behavioral checks; the paper does not evaluate such a product.

contradictions, limitations, and product threat

The abstract states: “achieving an average decrease of 67.52% in MiHN and 73.56% in MaHR compared to the RAG approach.” (p. 1). The RQ1 answer instead reports a 67.31% MaHR decrease (p. 7). Inference: These aggregate MaHR claims conflict; the paper does not explain the difference. The industry study uses 109 samples and PanguCoder-11B/34B (section 5.4, p. 9). Section 1 reports “an average decrease of 59.41% in MaHR and an average increase of 72.39% in EM, with only an average 0.031s overhead increase compared to the base model.” (p. 2). The accuracy and hallucination changes compare with RAG; the overhead compares with Base. These are reported results, not independently reproduced experiments.

The authors identify model selection, Java-only evaluation, prompt design, and automated benchmark curation as threats (section 6.2, p. 10). On curation, they state: “While automation enables scalability to handle extensive benchmarking tasks, it introduces potential concerns regarding dataset correctness and quality.” Inference: Automated labels and narrow task coverage limit how much confidence these results provide for verified Rust.

Inference: The threat to agent-written verified Rust is substantial for API hallucinations, because a nonexistent or wrong proof helper can make a generated proof fail or, worse, lead the model to alter a specification. Dependency mining and identifier masks directly reduce invented names, but the paper offers no evidence on Rust, Verus, semantic contracts, or adversarial repository changes.

techniques for the catalog

• hierarchical dependency mining: statically collect local calls and global files, then provide the structural context to generation
• dependency constrained decoding: mask API-name and parameter-pattern logits using a prefix tree built from valid declarations
• hallucination metrics: report micro hallucinated elements and macro fraction of APIs with any hallucination, with explicit denominators
• component ablation: remove local dependencies, global dependencies, and decoding constraints to measure each contribution Checked 2026-10-19 against the full paper by a Codex agent (mid-paper quote: “For example, on APIHulBench-F, removing global dependencies causes a 33.39% drop in average EM and a 186% rise in average MaHR across six LLMs.”): checked the full paper text, quotations, numbers, page and section references, and technique steps; corrected the abstract and metric quotations, motivation example, page references, prompt components, tokenizer handling, decoding stopping condition, and unsupported denominator claim, and marked analytical conclusions as inferences; did not inspect implementation code, validate benchmark labels, or reproduce experiments.

Last edited: 2026-10-19