Synchronous and Asynchronous RL for LLMs

Reinforcement learning for a language model is both a learning problem and a distributed-systems problem.

The learning problem asks:

• Which behavior should receive a reward?
• How should that reward change the model?
• How far may the updated policy move without becoming unstable?

The systems problem asks:

• How quickly can the system generate and verify rollouts?
• How should GPUs be divided between generation and training?
• What waits for what?
• How old may a rollout become before it is unsafe or wasteful to train on?

These concerns cannot be optimized independently. A system can keep every GPU busy while feeding the trainer data that is too stale to support stable learning. It can also preserve perfectly fresh data while leaving expensive hardware idle.

This page explains that trade-off.

Prerequisite

The page on RL Environments for LLM Agents explains policies, tasks, rewards, rollouts, verification, and environment scaling. Here, the environment and reward already exist. The focus is how rollout generation and policy updates are scheduled.

Begin With One Training Step

Suppose a language model is learning to solve reasoning problems.

For one training step, the system might:

1. select a batch of prompts
2. generate several candidate responses for each prompt
3. score or verify the responses
4. calculate the learning signal
5. update the model weights
6. make the updated model available for the next rollouts

flowchart LR
    P["Select prompts"] --> G["Generate rollouts"]
    G --> V["Verify and score"]
    V --> L["Calculate loss"]
    L --> U["Update policy weights"]
    U --> P

The policy is the model behavior produced by a particular version of its weights.

If the weights change after every optimizer step, the versions can be labeled:

policy v0 -> policy v1 -> policy v2 -> policy v3

This versioning matters once generation and training run at the same time.

Why Rollout Generation Often Dominates

Training a response requires forward and backward passes over tokens that already exist. Generating a response is sequential: the model predicts one token, appends it to the context, and then predicts the next.

A batch may also contain responses with very different lengths:

rollout A:  180 tokens
rollout B:  420 tokens
rollout C:  950 tokens
rollout D: 4,100 tokens

The longest response can take much longer than the median response. Tool-using agents add more variability because a rollout may wait on:

• a test suite
• an external API
• a browser or sandbox
• a reward model
• environment reset and cleanup
• several rounds of tool use

This creates a long-tail latency problem. Most work may finish quickly while a few stragglers determine when the whole batch can advance.

Synchronous RL

In a synchronous design, generation and training happen in distinct phases.

flowchart LR
    W["Policy weights v7"] --> S["Generate complete rollout batch"]
    S --> B{"Every required rollout finished?"}
    B -- "No" --> B
    B -- "Yes" --> R["Verify and score batch"]
    R --> T["Train on batch"]
    T --> N["Publish weights v8"]
    N --> S

The barrier marked “every required rollout finished” is the defining property.

What synchronization provides

Every rollout in the batch was generated by the same policy version. When training begins, the system knows exactly which policy produced the data.

That makes the learning semantics easier to reason about:

• batches are clearly associated with one behavior policy
• no queue of old rollouts accumulates
• weight synchronization occurs at an obvious boundary
• debugging and reproduction are simpler
• on-policy algorithms receive relatively fresh data

Where synchronization wastes time

Assume seven rollouts finish in 30 seconds and the eighth finishes in 90 seconds.

0s                       30s                                      90s
|-------------------------|-----------------------------------------|
seven rollouts complete   most generation capacity waits            last rollout completes

The training phase cannot start at 30 seconds even though most of the batch is available.

The problem becomes more expensive when generation and training share a GPU pool:

• during generation, training resources may be idle
• during training, generation resources may be idle
• at the end of generation, fast requests leave capacity unused while stragglers finish

Synchronous training is not necessarily slow at small scale. It becomes unattractive when rollout durations are highly variable, outputs are long, or the hardware fleet is large enough that barrier waiting is expensive.

Asynchronous RL

An asynchronous design decouples rollout generation from policy updates.

Generation workers continuously produce rollouts. Completed rollouts enter a queue. Training workers consume eligible data when they have enough for an update.

flowchart LR
    D["Task and prompt source"] --> S["Sampling workers"]
    S --> V["Verifier or reward workers"]
    V --> Q["Eligible rollout queue"]
    Q --> T["Training workers"]
    T --> W["New policy weights"]
    W --> S
    M["Version and staleness controls"] --> Q
    M --> S

Generation no longer needs to stop whenever training starts. Training no longer needs to wait for every rollout from a particular launch group.

This overlap can improve:

• accelerator utilization
• rollouts produced per hour
• optimizer steps completed per hour
• resilience to a few slow trajectories

It also changes the data seen by the trainer.

The Producer-Consumer View

The asynchronous pipeline can be understood as a queueing system.

The main producers are sampling and verification workers:

prompts -> generated responses -> rewards -> eligible rollouts

The main consumer is the trainer:

eligible rollouts -> training batches -> optimizer steps

Let:

lambda = eligible rollouts produced per second
mu     = rollouts consumed by training per second

The desired operating region is approximately:

lambda ~= mu

This is not an exact equality at every instant. Both rates fluctuate. It is a capacity-planning target.

If production is too slow

lambda < mu

The trainer drains the queue and waits for more data. Training GPUs become idle.

Possible causes include:

• too few sampling GPUs
• low inference batch size
• slow environments or verifiers
• long responses
• rate-limited tools
• repeated rollout failures

If production is too fast

lambda > mu

The queue grows. Rollouts wait through more policy updates before the trainer consumes them.

Consequences include:

• increasing policy staleness
• higher memory or storage pressure
• more rollouts discarded after exceeding an age limit
• wasted inference compute
• a larger gap between the behavior being sampled and the policy being trained

Maximum throughput therefore does not mean maximizing generation independently. The pipeline must be balanced.

What Policy Staleness Means

Every rollout is generated by a behavior policy.

Suppose a rollout was generated by policy v10. While it waited in the queue, the trainer completed three updates and reached policy v13.

rollout policy version:  v10
current train version:   v13
staleness:               3 updates

A common operational definition is:

staleness = current policy version - rollout policy version

The exact definition can vary by system. Some count optimizer steps, weight publications, or actor refreshes. Record which definition is used.

On-policy and off-policy data

Data is on-policy when it comes from the same policy being optimized, or from a sufficiently close snapshot for the algorithm’s assumptions.

Data becomes increasingly off-policy as the generating policy and current policy diverge.

This distinction is not binary in an implementation. A rollout one update old may remain useful. A rollout many updates old may be misleading or unstable. The acceptable lag depends on:

• the optimization algorithm
• learning rate
• clipping or trust-region controls
• how quickly the policy changes
• task and reward distribution
• response length
• model size

max_staleness = 4 is an implementation setting, not a universal law.

Why Stale Data Can Destabilize Learning

Policy-gradient methods compare the probability of an action under two policies.

A simplified importance ratio is:

ratio = probability under current policy / probability under behavior policy

For a token action a in state s:

r(theta) = pi_theta(a | s) / mu(a | s)

Where:

• mu is the behavior policy that generated the token
• pi_theta is the current policy being optimized

If the policies are close, the ratio is near 1.

If they have drifted apart, ratios may become very large or very small. A few samples can then dominate the gradient estimate, increasing variance and making optimization unstable.

PPO- and GRPO-style objectives commonly clip probability ratios or otherwise constrain policy movement. Clipping limits extreme updates, but it does not make arbitrary staleness harmless:

• heavy clipping can discard much of a sample’s learning signal
• old samples can reduce effective batch quality
• corrections rely on accurate behavior-policy log probabilities
• policy drift can still produce unstable or biased practical updates

The safe conclusion is:

Importance weighting can compensate for some policy mismatch, but bounded staleness remains an algorithmic requirement, not merely a queue-management preference.

A Rollout Can Contain More Than One Policy Version

Most asynchronous systems update an inference worker between complete requests. A request then has one behavior-policy version.

PipelineRL explores in-flight weight updates. An inference server can receive newer weights while a long response is still being generated.

An illustrative sequence might look like:

tokens 1-200:   policy v20
tokens 201-500: policy v21
tokens 501-730: policy v22

flowchart LR
    A["Begin response with policy v20"] --> B["Generate token segment"]
    B --> C["Receive weights v21"]
    C --> D["Continue same response"]
    D --> E["Receive weights v22"]
    E --> F["Finish response"]
    F --> G["Store per-token training metadata"]

This reduces the time for which long generations remain tied to old weights. It is also more complex than ordinary periodic actor refresh:

• tokens in one trajectory may have different behavior policies
• existing KV-cache state was produced using earlier weights
• per-token log probabilities and policy-version metadata must remain correct
• the trainer must interpret mixed-version data consistently

The PipelineRL paper reports results for this design. Those results establish that the approach can work under the paper’s workloads and configuration. They are not a guarantee that every model, reward, or cluster will obtain the same learning-speed improvement.

The Main Control Mechanisms

Asynchronous systems use several controls together.

1. Bound staleness

Reject or stop scheduling data that would exceed a chosen version lag.

accept if rollout_staleness <= max_staleness
discard otherwise

A lower bound improves freshness but may:

• discard more completed rollouts
• reduce trainer utilization
• require more frequent weight transfer

A higher bound keeps the pipeline busier but increases off-policy risk.

2. Limit in-flight work

Do not generate more outstanding rollouts than the trainer can consume while they remain eligible.

This is backpressure. It prevents generation workers from filling a queue with data that will be stale before use.

The experimental Hugging Face TRL asynchronous GRPO trainer exposes this idea through controls including:

• max_inflight_tasks
• max_staleness
• queue_maxsize
• weight_sync_steps

These names are useful examples of the concepts. Their defaults are library choices, not generally optimal settings.

3. Synchronize weights deliberately

More frequent synchronization keeps actors fresher but adds transfer and coordination overhead.

Less frequent synchronization reduces overhead but increases policy lag.

Measure:

• transfer duration
• inference pause time
• model version propagation delay
• failures or partial updates
• time until all actors acknowledge a version

4. Keep the queue bounded

A bounded queue makes overload visible and limits memory consumption. Queue depth should drive scheduling decisions rather than merely appear on a dashboard.

When the queue is near capacity, the system can:

• pause new rollout admission
• reduce concurrency
• prioritize short or high-value tasks
• drop data that is already too old
• add training capacity if the allocation permits it

5. Constrain policy movement

Systems controls do not replace learning controls.

Depending on the algorithm, stability may also require:

• a suitable learning rate
• probability-ratio clipping
• KL penalties or monitoring
• gradient clipping
• entropy monitoring
• reference-policy comparisons

Increasing max_staleness without reevaluating these controls changes the optimization problem.

Dividing GPUs Between Sampling And Training

Suppose a cluster has G GPUs:

G = G_sampling + G_training + G_other

G_other may include separate verifier, reward-model, reference-model, or orchestration resources.

The right split cannot be selected from GPU count alone. Measure both sides of the pipeline.

Sampling-side inputs

• model size and parallelism strategy
• prompt-length distribution
• response-length distribution
• number of responses per prompt
• inference batch size
• decoding settings
• KV-cache capacity
• environment and tool latency
• reward or verifier latency
• timeout and retry behavior

Training-side inputs

• optimizer batch size
• token count per batch
• forward and backward throughput
• gradient accumulation
• model and optimizer sharding
• checkpointing
• reference-policy or KL computation
• weight-gathering and transfer cost

The allocation objective is not:

maximize sampling tokens per second

It is closer to:

maximize useful policy improvement per unit of wall time and compute

That objective includes data freshness and learning stability.

Why Response-Length Distribution Matters

An average response length is not enough.

Consider two workloads:

Workload A: nearly every response is 800-1,000 tokens
Workload B: most responses are 300 tokens, but 5% exceed 8,000 tokens

They may have a similar mean while producing very different scheduling behavior.

For synchronous training, the upper tail influences barrier time.

For asynchronous training, long responses influence:

• how long a request remains attached to an old policy
• how much KV-cache memory it occupies
• how many shorter requests can be batched beside it
• rollout age on completion
• timeout and truncation rates

Profile at least:

• p50, p90, p95, and p99 completion length
• p50, p95, and p99 rollout duration
• length by task type and reward outcome
• truncation and timeout rates
• length changes over training

That final item matters because a policy may learn to produce longer responses. A layout calibrated before training can become unbalanced later.

Batch Size, KV Cache, And GPU Throughput

During autoregressive generation, the inference engine stores attention keys and values for tokens already processed. This KV cache avoids recomputing the entire prefix for every next token.

More concurrent or longer sequences require more KV-cache memory.

The generation batch therefore cannot grow without limit:

model weights + runtime memory + KV cache <= available GPU memory

At low batch sizes, decoding may underuse the GPU because model weights must be read for relatively little work. Increasing the batch can amortize that cost and improve throughput. Eventually, compute capacity or memory limits become the bottleneck.

This is related to the roofline view of GPU performance:

• a memory-bound workload is limited by data movement
• a compute-bound workload is limited by arithmetic capacity

The NVIDIA roofline documentation explains this distinction. The exact transition depends on the model, hardware, precision, inference engine, and sequence mix; it should be profiled rather than assumed.

Modern inference engines also schedule prefill and decode work dynamically. The vLLM optimization guide documents how batched-token limits trade throughput against latency and how combining compute-heavy prefill with memory-heavy decode can improve utilization.

Simulate Before Running The Full Job

Large RL runs are expensive experiments. A simple capacity model can reject obviously poor layouts before allocating the full cluster.

flowchart TD
    P["Profile representative workloads"] --> I["Fit sampling-throughput estimate"]
    P --> T["Fit training-throughput estimate"]
    P --> R["Measure response-length distribution"]
    I --> S["Sweep sampling and training GPU splits"]
    T --> S
    R --> S
    S --> M{"Fits memory limits?"}
    M -- "No" --> X["Reject layout"]
    M -- "Yes" --> L{"Meets staleness limit?"}
    L -- "No" --> X
    L -- "Yes" --> C["Estimate step time and utilization"]
    C --> B["Run a bounded pilot"]
    B --> V["Compare prediction with observed behavior"]
    V --> H["Select full-run configuration"]

What the model should estimate

• sampling tokens or rollouts per second
• training tokens or samples per second
• expected queue depth
• expected rollout age
• GPU utilization by role
• weight-transfer overhead
• memory use under representative sequence lengths
• sensitivity to longer responses or slower verification

What the model cannot prove

A systems simulation does not prove:

• that rewards are correct
• that the algorithm will converge
• that stale data remains statistically safe
• that the task distribution represents production
• that the policy will not learn longer or exploitative behavior

The simulation narrows the capacity-planning search. A real pilot still has to validate learning behavior.

A Small Illustrative Allocation Exercise

Imagine an eight-GPU run after accounting for any CPU-only services.

Start with:

3 GPUs for sampling
5 GPUs for training

During a pilot, observe:

rollout queue frequently empty
training GPU utilization low
sampling GPUs consistently busy
staleness near zero

The trainer is starved. Moving capacity toward sampling may help.

Now suppose a different split produces:

queue depth grows continuously
rollout staleness approaches the rejection limit
many completed rollouts are discarded
training GPUs consistently busy

Generation is overproducing relative to training. More sampling throughput is actively wasteful.

The useful split is the one that keeps both stages productive while satisfying memory and staleness constraints. It does not have to make both utilization graphs read 100%.

What To Monitor

Monitor systems health, data freshness, and learning quality together.

Area	Metrics	What they reveal
Sampling	tokens/sec, rollouts/sec, active sequences, GPU utilization	Whether inference capacity is productive
Tail behavior	p50/p95/p99 rollout duration and completion length	Whether stragglers or long reasoning dominate
Environment	reset, tool, sandbox, and verifier latency	Whether the model is actually the bottleneck
Queue	depth, enqueue rate, dequeue rate, oldest-item age	Whether production and consumption are balanced
Freshness	staleness histogram, accepted and discarded rollouts by version	How much useful inference work survives
Weight sync	frequency, duration, propagation lag, failures	The cost and reliability of actor refresh
Policy movement	KL divergence, importance-ratio distribution, clip fraction	Whether current and behavior policies are separating
Optimization	loss, gradient norm, overflow, entropy	Whether training remains numerically stable
Learning	train reward and fixed held-out reward	Whether policy quality is improving rather than only fitting training tasks
Behavior	completion length, tool calls, error and truncation rates	Whether the policy is changing its strategy or exploiting limits
Economics	GPU-hours, cost per eligible rollout, reward gain per GPU-hour	Whether higher throughput produces useful improvement

No single metric is enough.

For example:

• rising rollout throughput with rising discard rate may save no useful time
• rising training reward with flat held-out reward may indicate overfitting
• rising reward and response length may indicate reward exploitation
• high GPU utilization with unstable policy ratios may be a learning regression

Systems Efficiency And Learning Efficiency Are Different

Systems efficiency concerns how much work the infrastructure completes:

tokens per second
rollouts per hour
optimizer steps per hour
GPU utilization

Learning efficiency concerns how much useful policy improvement that work produces:

held-out reward gain per rollout
success-rate gain per optimizer step
quality gain per GPU-hour
quality gain per dollar

An asynchronous system can improve systems efficiency while reducing learning efficiency if its data is too stale. The final comparison should be made on a quality-versus-time or quality-versus-compute curve, not throughput alone.

When Synchronous Training Is A Good Choice

Prefer a synchronous design when:

• the system is still being debugged
• the hardware scale is modest
• responses are short and similar in length
• strict on-policy behavior is important
• the algorithm has not been validated under stale data
• reproducibility and simple failure analysis matter more than maximum utilization
• weight transfer or queue infrastructure would dominate the workload

Synchronous execution is also a useful baseline. An asynchronous system should demonstrate a better quality-versus-wall-time result against it.

When Asynchronous Training Is Worth Considering

It becomes more attractive when:

• rollout generation dominates end-to-end time
• response or environment latency has a long tail
• barriers leave substantial hardware idle
• the cluster is large enough for separate sampling and training pools
• the algorithm tolerates a measured amount of policy lag
• policy versions and behavior log probabilities are tracked correctly
• the system can enforce backpressure and discard stale work
• the team can operate distributed queues, weight transfer, and observability

Do not adopt it only because the architecture is more advanced.

A Practical Adoption Sequence

1. Make the synchronous baseline correct. Verify rewards, held-out evaluation, checkpoints, and policy updates.
2. Profile the complete pipeline. Include generation, tools, environments, rewards, training, and weight transfer.
3. Confirm that waiting is material. Measure barrier idle time and tail latency.
4. Introduce separate policy versions. Tag every rollout and preserve behavior-policy log probabilities.
5. Decouple generation and training with bounded queues.
6. Set a conservative staleness limit.
7. Add backpressure and stale-sample rejection.
8. Compare learning curves, not only throughput.
9. Increase concurrency gradually.
10. Recalibrate when the model, task mix, or response-length distribution changes.

Current Implementations

PipelineRL

ServiceNow/PipelineRL is an open-source asynchronous RL implementation with actor, verifier, preprocessing, training, streams, and in-flight weight-update components.

The associated PipelineRL paper is useful for understanding its algorithm-system design and experimental assumptions.

Treat PipelineRL as a concrete architecture to study or evaluate, not as a universal template. Its reported results depend on its models, tasks, cluster, optimization settings, and baselines.

Hugging Face TRL

TRL’s asynchronous GRPO trainer decouples generation on a vLLM server from training and includes explicit controls for:

• in-flight generation tasks
• rollout queues
• weight synchronization
• sample staleness

The API is under trl.experimental. Its documentation describes it as intentionally minimal rather than a general-purpose distributed RL platform.

vLLM

vLLM is an inference engine commonly used by RL training stacks for high-throughput generation. It provides controls for batching, scheduling, parallelism, and KV-cache memory.

An inference engine accelerates sampling. It does not by itself solve policy staleness, reward correctness, or training stability.

Common Mistakes

Treating every completed rollout as valuable

A completed rollout that is too stale to use is paid-for waste. Admission limits should consider likely consumption time.

Choosing the GPU split once

The right split can change as completion lengths, task mix, verifier cost, or policy behavior changes.

Using average latency

Barrier time and queue age are driven by distributions and tails, not only means.

Maximizing utilization without measuring quality

Busy hardware can produce stale or low-value data faster.

Assuming importance ratios remove all off-policy risk

High-variance ratios, clipping, approximate log probabilities, and policy drift still matter.

Generalizing a published speedup

A reported speedup belongs to its experimental setup. Reproduce the comparison on the target workload.

Optimizing infrastructure before reward quality

Asynchrony can accelerate a faulty reward loop. It cannot repair an invalid evaluator or unrepresentative task distribution.

Summary

Synchronous RL keeps rollout generation and training in lockstep. It offers clearer data provenance and simpler on-policy behavior, but long or uneven rollouts can leave expensive resources waiting.

Asynchronous RL overlaps generation and training. It can reduce idle time, but it introduces queues, policy lag, stale data, weight-transfer overhead, and a tighter connection between systems configuration and learning stability.

The central design problem is not:

How do we keep every GPU busy?

It is:

How do we maximize useful, stable policy improvement
while respecting compute, memory, and data-freshness limits?

References

• Efficient Reinforcement Learning - Rhythm Garg and Linden Li, Applied Compute
• PipelineRL paper
• PipelineRL repository
• Hugging Face TRL: Asynchronous GRPO
• vLLM: Optimization and Tuning
• vLLM: KV Cache Configuration
• NVIDIA Nsight Compute: Roofline Charts