Asynchronous RL training

Keep generating rollouts while the trainer updates the model. Asynchronous RL lets sampling work ahead by a bounded number of policy versions, which is useful when trajectories take different amounts of time or spend time waiting on tools.

2 of 5 · Previous: Synchronous training · Next: Policy versions and trajectory control

Start from the synchronous example

In the synchronous example, change two constants:

MAX_STALENESS = 2
CONCURRENCY = 192

MAX_STALENESS sets the trainer's policy-age bound; CONCURRENCY sets the engine's rollout concurrency. Schedule defaults to admit_ahead=2. The environment, reward, group size, and optimizer stay the same.

With 64 trajectories per training batch, 192 concurrent trajectories leaves room for the current batch and up to two more batches of work. These settings control client admission; they do not allocate more GPUs. Choose concurrency that your environment and available API capacity can use.

What changes in the loop

In synchronous mode, batch B begins sampling after batch A's optimizer update. In asynchronous mode, B can already be sampling while A is training. B may therefore contain responses from an older policy than the one used for its forward/backward computation.

Each sampled segment records the policy that generated it. With max_staleness=2, a segment generated by policy 10 can train against policy 10, 11, or 12. It cannot train against policy 13 under the default wait policy. The optimizer update following forward/backward advances the policy again.

Slow trajectories make the trainer wait

The default stale_policy="wait" holds back optimizer progress when an outstanding group would otherwise become too old. Sampling and tool calls continue during that wait. A long trajectory keeps its opportunity to train.

The bound applies to every generated segment, including early turns of a long conversation. A later turn sampled with newer weights does not make the earlier turns younger. The trainer may need to wait before the age limit is reached if several outstanding groups must still fit into the remaining updates.

Start with the wait policy. Masking old spans or allowing unbounded age changes which learning signal reaches training and requires explicit configuration.

Synchronous overlap is a different setting

You do not need policy staleness to overlap sampling with forward/backward. The synchronous recipe already supports that within one batch:

forward_backward_batch="auto" chooses the supported path. Leave it at its default when switching between these modes. Explicit ForwardBackwardBatch(...) settings are for synchronous group-centered overlap and cannot be combined with positive max_staleness.

For synchronous workloads, you can tune submission size separately from optimizer-batch size:

# AsyncTrainer option for a synchronous run:
forward_backward_batch = rl.ForwardBackwardBatch(
    min_sequences=32,
    min_tokens=65_536,
)

Either threshold triggers submission; complete groups stay together. Smaller chunks can begin earlier but create more requests. Batch-centered advantages (rl.Batchwise) need the full batch's rewards, so the automatic path waits for those rewards before forward/backward.

Keep one policy per trajectory to start

Async training still defaults to sampling_policy="trajectory": a rollout keeps the policy that started it, even while other groups train. Multi-turn rollouts can reuse their existing prefix under that policy.

A turn can span several sampling requests, or segments. With trajectory pinning, every segment uses the same policy. With sampling_policy="segment", later requests can use newer policies—even within the same assistant turn.

The policy and trajectory guide compares both modes visually, along with staleness handling and group completion options.

This is separate from the training-age limit. The trainer can be two versions ahead while a trajectory continues on its original policy. The wait policy prevents it from advancing beyond what the outstanding trajectories can use.

For very long conversations, you can instead let later turns use newer weights while retaining bounded-age KV. That is covered in Policy versions and trajectory control.

Measure the effect

Compare the two modes at the same task and batch size, using reward and evaluation quality alongside timing. Faster updates alone do not establish better training.

If staleness waits dominate, inspect long trajectories and tool latency. More lookahead cannot eliminate a wait while keeping the same age bound. Increasing max_staleness trades fresher data for more scheduling freedom; measure its effect on held-out reward.

step.metrics already includes sampling counters. Send that dictionary to your logger; you do not need to poll the engine separately. Sampling metrics cover the interval since the previous result, including work on future batches. Reward and training metrics describe the batch being consumed. Evaluation uses its own engine and is recorded separately.

Oversampling

Schedule(oversample_factor=1.5, ...) admits extra whole groups so the trainer has more completed groups to choose from. Surplus groups remain eligible for later updates. They share the same concurrency and staleness limits, so oversampling does not create an unlimited queue or bypass the wait policy.

Tune it after measuring the basic async recipe, especially if groups have widely varying completion times.

Next: Compare policy and trajectory controls →