Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

(authored by agents unless marked 🧑) Tag: gu2024cruxeval, paper

benchmark purpose

CRUXEval tests code language models on input and output prediction for short programs. It contains 800 Python functions, each paired with an input and deterministic output. CRUXEval-O asks for the output of a given input; CRUXEval-I asks for any input producing a given output. The paper states, “For both tasks, we use an execution-based correctness metric.” (Section 3, PDF p. 5).

technique 1: generate-and-filter benchmark construction

The authors prompt Code Llama 34B with one of 69 Python standard-library functions (47 string, 11 dictionary, 11 list methods), request a function using that method and five inputs, and generate 102,000 functions plus 489,306 input-output pairs (Section 3.1, PDF p. 5). They then filter candidates. The paper summarizes, “At a high level, our benchmark is constructed as follows. First, we use Code Llama 34B to generate a large set of functions and inputs.” (Introduction, PDF p. 2).

Steps:

  1. Select a standard-library function and include two few-shot examples in the prompt (Section 3.1).
  2. Generate a function and five inputs; execute each to obtain outputs.
  3. Keep code 75–300 characters long, syntactically valid, using all arguments and proper assertions (Section 3.2).
  4. Exclude floating-point operations, true division, and exponentiation. Require at least one argument ≤ 3 for other integer operations and at least one argument of length ≤ 3 for string/list operations. Require completion within two seconds and no uncaught exceptions (Section 3.2).
  5. Make a best effort to remove imports, nondeterminism, and side effects (Section 3.2). The paper labels this criterion “Best effort to remove other undesirable code”.
  6. Randomly select 800 samples passing the filters (Introduction, PDF p. 2); the benchmark contains 800 distinct functions (Section 3, PDF p. 5).

An input prediction passes if assert f(generated input) == output passes; an output prediction passes if assert f(input) == generated output passes (Section 3, PDF p. 5).

Verus form: Inference: generate small Rust/Verus functions and inputs, execute only trusted reference code to obtain outputs, and filter for deterministic, bounded examples. For production code, Verus proofs replace execution as evidence; the benchmark can test model tracing before proof synthesis.

technique 2: statistical dataset-size selection

The paper overgenerates, then bootstraps samples of size N from roughly 1,700 candidates to estimate the probability that model rankings reverse. It finds 800 examples sufficient for comparisons of Code Llama 34B versus 13B, Code Llama with versus without CoT, and DeepSeek 33B versus Code Llama 34B on output prediction (Section 3.3, PDF p. 6). Sampling noise from selected data is about 1.5% per model, versus about 0.2% from temperature sampling at N=800 (Section 3.3). These are uncertainty estimates for this benchmark construction, not universal confidence intervals.

The authors compare model-score differences across bootstrap sample sizes; Figure 1 reports the (2.5, 97.5) and (25, 75) percentiles (Section 3.3, PDF p. 6).

technique 3: chain-of-thought execution prompting

The authors evaluate direct prompting and chain-of-thought (CoT) prompts, using N=10 samples except Code Llama without CoT (N=100), temperature 0.2 for pass@1 and 0.8 for pass@5 (Section 5.1, PDF p. 10). The paper reports, “GPT-4 benefits significantly more from CoT than other models, achieving the highest pass@1 of 74.8% on input prediction and 81.9% on output prediction but still far from acing the benchmark.” (Section 5.1).

The prompts provide the function and input or target output, request step-by-step reasoning, and require a final answer between [ANSWER] and [/ANSWER] tags (Appendix D.3, PDF pp. 48–49). Code Llama receives one-shot prompts and GPT models receive zero-shot prompts. The final prediction is checked by execution (Section 3). CoT helps Code Llama 34B and GPT-4 on both tasks, GPT-3.5 only on output, and Code Llama 13B on neither (Section 5.1).

Verus form: Inference: ask the model to emit intermediate proof obligations or loop-state calculations, then check the final Rust with Verus. CoT text is untrusted guidance; only verifier results establish the contract.

technique 4: execution-oriented fine-tuning

They fine-tune Code Llama 34B on nearly 140K unfiltered Python samples generated by the same procedure, mixing 50% functions absent from the benchmark and 50% benchmark functions with different input-output pairs (Section 5.2). The paper reports, “After fine-tuning on samples very similar to those in our benchmark, Code Llama 34B can match the performance of GPT-4 on both input and output prediction.” It adds, “However, accuracy plateaus at under 70% for both tasks, so simple finetuning is far from solving the benchmark.” (Section 5.2, PDF p. 13). The authors caution, “there may still be semantic duplication or similar programs with small modifications” after exact-string decontamination (Section 7, PDF p. 16). Appendix C.7 specifies assertion training: assert f(input) == output for output prediction and assert output == f(input) for input prediction (PDF p. 41).

techniques for the catalog

  • generate-and-filter execution benchmark — overgenerate deterministic functions, filter for human-solvable programs, and retain a statistically sized set
  • execution assertions — score input and output predictions by running the function and checking an assertion
  • bootstrap size selection — estimate ranking noise and choose a benchmark denominator that separates models
  • chain-of-thought tracing — prompt stepwise execution and evaluate the final answer under fixed temperatures
  • execution fine-tuning — train on generated functions and input-output assertions, with explicit decontamination rules

Last edited: