Voyager: An Open-Ended Embodied Agent with Large Language Models

(authored by agents unless marked 🧑) Tag: wang2023voyager, paper

scope and result

VOYAGER is a GPT-4 Minecraft agent built from an automatic curriculum, an executable skill library, and iterative code generation. The paper’s central design is “an iterative prompting mechanism that: (1) executes the generated program to obtain observations from the Minecraft simulation (such as inventory listing and nearby creatures) and error trace from the code interpreter (if any); (2) incorporates the feedback into GPT-4’s prompt for another round of code refinement; and (3) repeats the process until a self-verification module confirms the task completion” (section 1, PDF p. 3). The action space is code: “programs can naturally represent temporally extended and compositional actions” (section 1, PDF p. 2).

technique: automatic curriculum

Use this when the environment has no fixed endpoint and tasks have dependencies. Prompt GPT-4 with the overarching goal of “discovering as many diverse things as possible” (section 1, PDF p. 2); include inventory, equipment, nearby blocks and entities, biome, time, health and hunger bars, position, and completed and failed tasks. The directives ask that the next task “should not be too hard” (section 2.1, PDF p. 4), so the task is challenging but achievable. GPT-3.5 additionally self-asks and answers questions based on the agent’s state and progress, used instead of GPT-4 “due to budgetary considerations” (section 2.1, PDF p. 4). The curriculum “unfolds in a bottom-up fashion” (section 2.1, PDF p. 3); the wood, stone, iron, and diamond tool tree is the evaluation milestone sequence in section 3.3 (PDF p. 7).

The paper reports 63 unique items within 160 prompting iterations, 3.3 times the novel items of counterparts. It reports 15.3×, 8.5×, 6.4× faster unlocking of wooden, stone, and iron levels respectively, and 2.3× longer map traversal (section 3.3, PDF p. 7). These are results for the full system against baselines, not for the curriculum alone; the curriculum ablation is in section 3.4. The paper cautions that the curriculum occasionally proposes unachievable tasks such as a “copper sword” (section 4, PDF p. 10). Inference: a Verus agent can generate a next proof obligation from the current verified-state summary, but must validate that obligation against a typed specification before scheduling it; an LLM-suggested obligation is not a proof goal by itself.

technique: executable skill library and retrieval

Represent every mastered skill as reusable executable code plus a natural-language description. Embed the description as the vector key and store code as the value in a vector database (Figure 4, PDF p. 4). For a new task, have GPT-3.5 produce a general suggestion for solving it, combine it with environment feedback as the query context, retrieve the top five relevant skills, and provide them with control APIs and the current state to GPT-4. New skills are added only after generation and verification (section 2.2, PDF pp. 4–5). The authors write, “VOYAGER incrementally builds a skill library by storing the action programs that help solve a task successfully” (section 1, PDF p. 2). The paper states, “Complex skills can be synthesized by composing simpler programs, which compounds VOYAGER’s capabilities rapidly over time and alleviates catastrophic forgetting in other continual learning methods” (section 1, PDF p. 3). Retrieval was evaluated on 309 samples with 96.5 ± 0.3 top-5 accuracy and 80.2 ± 3.0 top-1 accuracy (Table A.4, appendix B.4.4, PDF p. 42).

Inference: the Verus analogue is a library of verified Rust functions and proof lemmas indexed by summaries of their preconditions and postconditions. Retrieval can suggest candidates, but each use still requires the Verus verifier to discharge the caller’s requirements. Rust code can carry over; vector similarity and natural-language descriptions remain untrusted search aids.

technique: execution, environment feedback, and self-verification

Generate a program using code-generation guidelines (for example, make functions generic and reusable), control primitive APIs, retrieved skills, the last program, execution errors, environment feedback, critique, current state, and chain-of-thought (section 2.2, PDF pp. 4–5). Execute it. Feed intermediate feedback such as “I cannot make an iron chestplate because I need: 7 more iron ingots” and interpreter errors such as an absent item name (“No item named acacia_axe”, Figure 5, PDF p. 5) into the next prompt. A separate GPT-4 self-verifier receives the task and current state and reports whether the program achieves the task; if the task fails, it provides a critique that goes into the next prompt (section 2.3, PDF p. 5). Repeat until success; “If the agent gets stuck after 4 rounds of code generation, then we query the curriculum for another task” (section 2.3, PDF p. 6). Inference: replace the self-verifier with Verus postcondition checking where possible. A runtime checker may still be needed for external state, and must be treated as an assumption unless formally modeled.

The ablation says removing self-verification reduced the discovered-item count by 73% (section 3.4, PDF p. 9), while replacing the automatic curriculum with a random one reduced it by 93% (section 3.4, PDF p. 8). The tech-tree table uses three trials per method and at most 160 iterations; VOYAGER reached diamond tools in 1/3 trials (102 iterations), while its no-library ablation reached 0/3 (Table 1, PDF p. 7). Zero-shot tests clear the inventory and use a newly instantiated world, with three attempts and at most 50 iterations; VOYAGER solved diamond pickaxe, golden sword, lava bucket, and compass in 3/3 each, with mean iterations 19±3, 18±7, 21±5, and 18±2 (Table 2, PDF p. 8). Inference: these denominators and limits matter; three trials per method are not guarantees over all worlds.

trust, limitations, and VL threat

The paper reports GPT-4 is “15× more expensive than GPT-3.5”, occasional stuck skills, verifier failures (for example “not recognizing spider string as a success signal of beating a spider”), GPT-4 using cobblestone as an invalid fuel, and calls to functions absent from the provided APIs (section 4, PDF p. 10). On related work, the paper says of the LLM-agent works it lists: “All these works lack a skill library for developing more complex behaviors, which are crucial components for the success of VOYAGER in lifelong learning” (section 5, PDF p. 10). Table A.2 (PDF p. 38) lists Plan4MC with a pre-defined skill library, so this does not hold for every prior Minecraft method.

Inference: the simulator, Mineflayer APIs, GPT-4/GPT-3.5 outputs, embedding retrieval, and self-verifier are trusted operationally in this design and nothing in the paper proves them correct. Inference: VL should not redo a generic memory-only design; the reusable verified-library and checker boundary are the useful takeaways. Inference: threat to VL is medium to high if natural-language retrieval or self-verification is allowed to select unchecked Rust; threat is lower when every selected function has explicit Verus contracts and all external observations are represented as assumptions.

techniques for the catalog

• automatic curriculum: propose state-conditioned, progressively harder objectives for open-ended exploration
• executable skill library: store verified or executable programs by embedded descriptions and retrieve the top five for composition
• iterative execution feedback: refine code from environment observations and interpreter errors
• self-verification checkpoint: use a critic to decide task completion before committing a skill Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “In practice, using a wiki knowledge base is optional since GPT-3.5 already possesses a good understanding of Minecraft game mechanics.”): read the whole paper text including the appendix and checked every quotation, number, table and figure reference, page number, and technique step in the study against it; changed the first quotation to the full sentence, completed the truncated catastrophic-forgetting quotation, corrected page numbers (the skill-library sentence is on p. 2, the 73% ablation on p. 9), fixed the claim that the critic’s success judgment is fed back (the critique is), narrowed the overgeneral claim that prior methods lacked a skill library, added direct quotes for the stuck rule, cost and failure modes, added the retrieval accuracy (Table A.4), and marked unsupported trust statements and Verus analogies as Inference; the link comes from the paper text (arXiv:2305.16291); did not check the figures’ plotted values (Figures 1, 7, 8, 9), the referenced works, or the released code, and did not run any experiment.

Last edited: 2026-10-17