Code World Models

A language model can learn a great deal about programming from source code. It can recognize syntax, common algorithms, library conventions, error patterns, and the kinds of changes developers usually make.

But source code is a description of a program. Execution is what happens when that description runs with a particular input inside a particular environment.

During execution:

variables acquire and change values
conditions select one path while skipping others
functions create and return data
files, databases, and external services may change
exceptions interrupt the expected flow
commands alter the surrounding system

A code world model is trained to represent and predict some of these changes. Instead of learning only which code token is likely to appear next, it also learns patterns such as:

current program state + next action -> likely next program state

This does not turn a language model into a real Python interpreter, compiler, or operating system. It gives the model a learned approximation of how computational environments behave.

First: What Is A World Model?

The term world model comes from research on agents and model-based reinforcement learning. An agent observes an environment, takes an action, and receives a new observation. A world model learns the environment’s dynamics well enough to predict what may happen next.

flowchart LR
    A["Current state"] --> B["Action"]
    B --> C["Environment transition"]
    C --> D["Next state"]
    D --> E["New observation"]

For a robot, the state might include its location and the positions of nearby objects. An action might be moving one arm. The next state describes where the arm and objects are afterward.

For software, the “world” is computational:

World-model concept	Software example
State	Variables, call stack, memory, files, repository contents, process state
Action	Execute a line, call a function, edit a file, run a command, invoke a tool
Transition	The change caused by that action
Observation	Return value, printed output, exception, compiler message, test result
Environment	Interpreter, container, repository, terminal, operating system, connected services

The world-model idea and the language-model architecture are not opposites. An autoregressive language model can be trained to predict a sequence containing states, actions, and observations. In that case, the language model is one implementation of a world model.

Static Code And Running Code Are Different Evidence

Consider this small program:

balance = 100
fee = 5

if balance >= fee:
    balance = balance - fee

The source text tells us which operations exist. An execution trace records what occurred for this run:

Step	Executed action	Relevant state afterward
1	`balance = 100`	`balance = 100`
2	`fee = 5`	`balance = 100`, `fee = 5`
3	evaluate `balance >= fee`	condition is `True`
4	`balance = balance - fee`	`balance = 95`, `fee = 5`

Change the starting balance to 2 and the fourth step is not executed. The same program structure can therefore produce a different trace for a different input.

This distinction matters:

Static source-code training	Execution-aware training
Learns from code as text	Learns from actions and resulting states
Sees all written branches	Sees the path taken for a particular run
Can infer likely behavior	Receives direct examples of observed behavior
Captures syntax and coding conventions well	Provides a stronger signal for state changes and control flow
Does not require the code to run	Requires executable programs and controlled environments

Ordinary code models are not ignorant of semantics. Source code, tests, documentation, bug fixes, and repeated programming patterns let them infer substantial behavioral knowledge. Execution data adds a more direct training signal; it does not create semantics from nothing.

What Is In An Execution Trace?

An execution trace is an ordered record of events from a program run. Depending on the tracing system, a frame can include:

the source line or command being executed
function calls and returns
local variable names and values
exceptions
selected memory or object state
terminal commands and their output
changes to files or repository state

A simplified trace may look like this:

CALL calculate_total(items=[10, 20])
LINE total = 0
STATE total=0
LINE total += item
STATE total=10, item=10
LINE total += item
STATE total=30, item=20
RETURN 30

Training on many such sequences gives the model examples of the relationship between an action and its consequence.

There are two related meanings of trace in coding-agent systems:

Trace type	What it records	Primary use
Program execution trace	Lines, calls, values, memory, returns, and exceptions inside a running program	Learning or inspecting program behavior
Agent trajectory	Model messages, tool calls, terminal output, file edits, tests, and later decisions	Learning or evaluating how an agent works

A coding model can be trained on both. Program traces teach lower-level execution dynamics. Agent trajectories teach interaction with repositories, terminals, tests, and other tools.

How Execution-Aware Training Works

At a high level, building a code world model requires more than collecting source files:

flowchart TD
    A["Collect programs,<br/>repositories, and tasks"] --> B["Build controlled,<br/>executable environments"]
    B --> C["Run programs and<br/>agent actions"]
    C --> D["Record states, actions,<br/>outputs, and failures"]
    D --> E["Train the model to predict<br/>transitions and behavior"]
    E --> F["Supervised instruction<br/>fine-tuning"]
    F --> G["Reinforcement learning on<br/>verifiable tasks"]
    G --> H["Evaluate simulation,<br/>coding, and agent behavior"]

1. Make the program executable

Public code is not automatically runnable. A repository may require:

a particular operating-system image
language and package versions
dependencies
build tools
environment variables
test data
setup commands

Container images are useful because they package much of this environment into a reproducible unit.

2. Run diverse inputs and actions

One execution covers only one path. Useful training data therefore needs many programs, inputs, mutations, commands, test runs, and failure cases.

For repository-level work, the system may apply a proposed change and then run:

a compiler or type checker
unit and integration tests
linters
application commands
continuous-integration steps

The resulting observations show which actions repaired the system, which actions caused new failures, and which actions had no useful effect.

3. Serialize the experience

The states and actions must be represented in a sequence the model can learn. For example, special tokens can distinguish:

a function call
a source line
local variables
a return
an exception
an agent command
environment feedback

This representation is a design choice. Recording every byte of memory would be impractical, while recording too little state may omit the relationship the model needs to learn.

4. Continue training

A model can first learn general language and code, then receive additional mid-training on execution and environment trajectories. Supervised fine-tuning can teach task formats and instruction following. Reinforcement learning can then reward results that are externally verifiable, such as:

tests passing
a compiler accepting the program
a generated answer matching a known result
a patch resolving a reproducible issue

Verification is important because plausible explanations of execution are not necessarily correct execution predictions.

How A World Model Could Help A Coding Agent

A normal coding agent already has access to a real computational environment. It can edit a file, run a command, read the result, and revise its plan.

decide -> act in real environment -> observe -> revise

A learned world model adds a possible internal planning step:

decide -> predict likely outcomes -> choose an action
       -> act in real environment -> observe -> revise

flowchart TD
    A["Repository state<br/>and task"] --> B["Generate candidate actions"]
    B --> C["Predict likely effects<br/>with learned world model"]
    C --> D["Select an action"]
    D --> E["Execute in the real<br/>tool or sandbox"]
    E --> F["Observe actual output,<br/>tests, and state"]
    F --> G{"Goal reached?"}
    G -- "No" --> B
    G -- "Yes" --> H["Return verified result"]

This may help when the model must choose among several expensive or risky actions. It can reason about likely consequences before spending time executing each one.

Possible benefits include:

Better state tracking

Long functions, nested calls, mutation, and loops require tracking how values change. Exposure to execution traces may improve this ability.

Predicting side effects

An edit can affect callers, tests, generated files, or system state. A world-model-oriented agent may become better at anticipating those effects before making the change.

Debugging

Given a failure and partial state, the model may predict which earlier transition likely went wrong or what value would produce the observed exception.

Filling in behavioral gaps

If some input, implementation, or intermediate state is missing, a model can generate plausible possibilities consistent with the surrounding program.

Planning tool use

An agent may estimate whether it should inspect a file, run a focused test, build the entire project, or gather another observation first.

These are research goals, not guarantees. The central question is whether a model trained on computational transitions can plan more effectively than one trained mainly on static code and ordinary agent trajectories.

A Neural Debugger Is A Prediction Tool

Meta’s CWM repository includes a demonstration described as a neural debugger. It presents a debugger-like interface in which the model predicts line-by-line execution and program state.

That looks similar to a conventional debugger, but the source of its answers is different:

Conventional debugger	Neural debugger
Executes the actual program	Predicts what execution would likely do
Reads real runtime state	Generates likely state
Exact within the debugger and runtime’s own limits	Probabilistic and can hallucinate
Requires an executable environment	May reason about incomplete or non-executed code
Reports the selected run	Can generate different plausible missing inputs or paths

The neural approach is useful precisely when actual execution is unavailable, incomplete, expensive, or being considered hypothetically. When the program can be run safely, the real debugger remains stronger evidence.

Prediction Is Not Execution

A model can produce a convincing trace that never occurred.

Prediction errors become more likely with:

unfamiliar libraries or language features
native extensions and undefined behavior
concurrency, timing, and race conditions
network and distributed-system behavior
hidden global state
reflection or dynamic code generation
environment-specific configuration
long traces in which small errors accumulate
external services whose state the model cannot observe

This creates model drift within the imagined execution. If one predicted variable is wrong, later predicted branches and outputs may also be wrong.

The safe hierarchy is:

use the model to generate hypotheses and prioritize actions
execute the selected action in an isolated real environment
treat compiler output, tests, runtime observations, and system state as evidence
revise the model’s plan when prediction and reality disagree

A world model can reduce blind trial and error. It should not remove the verification loop.

It Does Not Solve The Halting Problem

The halting problem asks whether a general algorithm could correctly decide, for every possible program and input, whether that program eventually stops or runs forever.

Turing’s result establishes that no such general decision procedure exists.

A learned model can still make useful narrower predictions:

classify whether familiar programs are likely to terminate
detect common infinite-loop patterns
estimate whether a bounded run will finish within a time limit
predict termination on the distribution represented in its training data
suggest invariants or ranking functions that a separate verifier can check

These are heuristic or restricted tasks. They do not decide termination for every arbitrary program.

The distinction is the same as weather forecasting: a model can make useful predictions without providing a proof. For program behavior, high predictive accuracy on a benchmark is not a way around undecidability.

CWM: A Current Research Example

Code World Model (CWM) was released by Meta FAIR in 2025 as a research model for studying code generation with world models.

As documented in its current model card:

Property	CWM
Architecture	Dense, decoder-only autoregressive language model
Size	32 billion parameters
Maximum training context	131,072 tokens
General training stages	Pre-training, execution-aware mid-training, supervised fine-tuning, and verifiable reinforcement learning
Execution data	More than 200 million memory traces of Python programs run in containers
Executable repositories	More than 30,000 repository container images
Agent data	3 million simulated agent-environment trajectories
Released artifacts	Pre-trained, SFT, and post-trained model checkpoints
Intended use	Noncommercial research involving code synthesis and understanding
Production status	Research-only; not evaluated or intended for production or general chatbot use

The exact counts describe CWM’s reported training mixture, not a minimum recipe every code world model must follow.

Open weights is not the same as unrestricted open source

The CWM repository releases its code under BSD-3. The model weights use Meta’s separate FAIR Noncommercial Research License, which excludes commercial use.

Calling CWM open-weights is therefore more precise than implying that the complete model can be used without material restrictions.

Running it requires substantial hardware

The model card says CWM can run on one GPU with 80 GB of VRAM when quantized. The repository’s default evaluation and demonstration configurations require 160 GB of combined GPU memory and RDMA-capable infrastructure.

These are different claims:

minimum-style inference configuration: quantized model on an 80 GB GPU
default research evaluation/demo configuration: multiple GPUs totaling 160 GB

CWM is not currently a lightweight local coding assistant for an ordinary laptop.

What The Report Actually Demonstrates

It is tempting to interpret every coding-benchmark result as evidence that execution simulation made the agent better. The CWM report’s own ablation is more nuanced.

In an 8-billion-parameter experimental setting:

adding Python execution traces substantially improved the reported program-input and program-output prediction results
those traces did not improve the reported SWE-bench-related metrics in that particular ablation
adding agent-environment trajectory data improved the agentic trajectory likelihood and SWE-bench result

This supports two separate conclusions:

execution-trace training can improve explicit execution prediction
better execution prediction does not automatically imply better end-to-end repository-agent performance

Agent success also depends on tool use, search, planning, context management, verification, and recovery from failure. World modeling may contribute to those abilities, but a final benchmark score reflects the whole system.

The released CWM results are best read as an early research case study, not proof that world-model training has replaced ordinary coding-agent design.

Important Design Questions

Code world models create several practical research questions.

Which state should be recorded?

Local variables are useful but incomplete. Files, heap objects, processes, databases, network responses, and environment variables may determine behavior. Recording more state increases cost and may expose sensitive data.

How broad must execution coverage be?

One run reveals one path. Training data needs variation across inputs, branches, failures, languages, libraries, repositories, and runtime versions. Otherwise the model may memorize common traces without learning robust transition patterns.

When is simulation cheaper than execution?

For a small Python function, running the program is usually faster and more reliable than asking a 32-billion-parameter model to imagine it. Learned prediction becomes more interesting when:

the environment is unavailable
setup is expensive
an action is destructive
many candidate plans must be screened
the model is filling in missing state
execution is slow, distributed, or difficult to reproduce

Even then, the cost of an incorrect simulated result must be considered.

How should predictions be calibrated?

An agent needs to know when its internal prediction is weak. Useful systems may require:

uncertainty estimates
multiple predicted outcomes
a threshold that triggers real execution
disagreement checks between models
short prediction horizons
automatic comparison between predicted and actual state

Without calibration, a more detailed imagined trace can create unjustified confidence.

Can the model learn unsafe actions?

Execution and agent trajectories may contain destructive commands, vulnerable code, secrets, or behavior that should not be reproduced. Data filtering does not replace runtime isolation. Model-generated code and commands must still run with bounded permissions, time, memory, filesystem access, and network access.

When This Approach Is Most Useful

Code world modeling is a promising fit for:

research on program-state tracking
learned execution prediction
debugging incomplete programs
ranking possible coding actions before execution
generating hypotheses for test failures
studying agents in controlled computational environments

It is a weaker fit when:

exact execution is cheap and available
a formal proof is required
behavior depends on hidden or rapidly changing external state
the program uses concurrency or hardware behavior the model cannot represent reliably
the model’s inference cost exceeds the cost of simply running the program

The goal is not to replace interpreters, compilers, debuggers, tests, or formal verification. It is to give an agent a better predictive model for deciding what to inspect or try next.

Key Takeaways

A world model predicts how an environment changes after an action.
In software, the state can include variables, memory, files, repository contents, processes, and tool output.
Static code and execution traces provide different training signals.
Program traces record runtime transitions; agent trajectories record interaction with tools and environments.
Execution-aware training can improve learned simulation, but simulation remains probabilistic.
A neural debugger predicts runtime state; a conventional debugger observes real runtime state.
Real execution and verification remain the source of truth.
Predicting likely termination on bounded examples does not solve the general halting problem.
Meta’s CWM is an open-weights, research-only case study rather than a production coding assistant.
Better execution prediction and better end-to-end coding-agent performance must be evaluated separately.