Learning from rewards

Reinforcement learning (RL) trains an LLM on the outcomes of its own attempts. You supply tasks and a scoring rule. The model generates responses or acts through tools in an agent loop. You score those attempts, then train the model toward behavior that earns higher rewards.

This chapter follows one batch from attempts to an update. The first RL run implements the loop with river_client.rl.

Demonstrate an answer or judge an attempt

In SFT, you supply a desired response. In RL, the model supplies the response and you judge it. This is useful when you can verify success more easily than you can write every step of a good solution.

A math problem can have a checkable final answer. A coding task can have tests. An agent can be scored on whether it completed a task in an environment. Each scoring rule is an imperfect representation of what you care about; the model learns from the rule you actually implement.

An RL policy is the model's distribution over actions given its context. For a language model, actions include generated tokens and the tool calls they form. A rollout or trajectory is one attempt: the generated response, or a sequence of model actions and environment observations.

The learning loop

The objective is to increase expected reward: average task success under the policy's sampling distribution and the task distribution you train on. A single successful response is evidence about one attempt, not proof that the model can reliably solve that kind of problem.

Sampling multiple responses exposes differences in outcomes. If every attempt is identical, comparing them gives little information about which behavior is better. Exploration must still produce useful attempts: random-looking text is not a substitute for a capable starting model.

From rewards to advantages

Consider four responses to the same question. Three are wrong and one is right. A simple verifier gives reward 0 or 1. The group's mean reward is 0.25.

An advantage measures how an attempt compares with a baseline. For unstandardized, group-centered advantages, subtract the mean reward of that question's group from each reward:

Positive advantage encourages the sampled actions; negative advantage discourages them relative to their alternatives. This describes the objective's local incentive, not a guarantee that every token's probability changes in that direction after a shared-parameter update.

rl.GroupCentered() provides this unstandardized centering in the introductory recipe. It draws on the group-relative approach introduced in DeepSeekMath. Recipes differ in whether they also divide by a reward standard deviation, how they normalize token losses, and which objective they optimize. “GRPO-style” does not uniquely specify those choices.

When a group has no signal

If all four rewards are 0, all four advantages are 0. The same is true if all four rewards are 1. A uniformly failing group and a uniformly successful group have very different quality, but neither gives this estimator a within-group preference.

Inspect reward/zero_variance_group_frac alongside mean reward. If all rewards are zero, first check the verifier, output parsing, token budget, and task suitability. More updates cannot extract a relative signal from identical scores.

From advantages to an update

The reward is a score for the attempt. The training objective operates on the model's probabilities for the tokens it generated. In a simple outcome-reward recipe, each generated token in an attempt receives that attempt's advantage. Prompt tokens and environment observations provide context without being reinforced as model actions.

This is a coarse form of credit assignment. A correct final answer does not tell you which reasoning step caused success. A successful trajectory may also contain unnecessary or unhelpful actions. Better tasks, rewards, and evaluation help you distinguish useful behavior from shortcuts.

The RL library records the original generated token IDs and their log probabilities. A log probability is the logarithm of the probability assigned to a sampled token. Keep this record tied to the policy that generated it; retokenizing visible text is not a reliable way to reconstruct the original training sequence.

Why keep the old probabilities?

The policy used for training can differ from the one that generated a rollout. An importance ratio compares the new and old probabilities of the same sampled token under its recorded context:

A ratio of 1 means they agree at that position. A ratio of 2 means the training policy assigns twice the probability to that token. Ratios help form objectives that account for sampling under a different policy; they do not make arbitrarily old data safe to use.

PPO and CISPO limit updates in different ways. PPO clips its surrogate objective; CISPO caps the importance weight used with the log-probability gradient. The first recipe uses CISPO. Start with the working recipe, then use Loss functions and Build your own RL system when you need to control the objective directly. The PPO paper develops the clipped surrogate.

Design a reward you can trust

Test the scoring function on known successes, failures, malformed responses, and plausible shortcuts before spending a training budget. In the math recipe, reward depends on both a correct number and the expected output format. A parsing failure can therefore look like a reasoning failure in the aggregate score.

Keep the reward aligned with the real task. If you reward brevity, inspect whether the model is omitting necessary work. If you use tests, inspect whether they cover the behavior you intend. If a model judges responses, audit examples where that judge disagrees with your own assessment.

Measure progress outside the training loop

Keep generation settings comparable, including temperature, token budget, and number of attempts. Report those settings alongside scores. Learning to answer within a tight budget can be valuable, but it is a different claim from improving performance at an unrestricted budget.

Try it: change the four rewards above to [1, 1, 0, 0], then to [1, 1, 1, 1]. Compute the advantages. Explain why mean reward rises while the within-group signal eventually disappears.