Loss functions
Choose a built-in objective below, or write a custom PyTorch loss
for RL, distillation, or supervised learning. Custom losses run on River's
training backend and have their own guide, with examples and AsyncTrainer
integration.
Built-in losses
Pass the loss name as loss_fn to forward_backward, with loss-specific
keyword arguments, for example:
model.forward_backward(data, loss_fn="cispo", eps_max=8.0).
Cross-entropy and the four RL objectives below sum token losses. To exclude a token, set its
weightsvalue to0.0forcross_entropy, or itsadvantagesvalue to0.0for an RL loss. Mask prompt tokens this way to train only on the completion.opsduses a different reduction, described below.
loss_fn | Use | Required per-datum fields | Arguments (default) |
|---|---|---|---|
cross_entropy | Supervised fine-tuning | input_ids, weights (target_tokens optional) | — |
importance_sampling | Off-policy policy gradient | input_ids, old_logprobs, advantages | — |
ppo | Clipped policy optimization | input_ids, old_logprobs, advantages | clip_low (0.2), clip_high (0.2) |
cispo | Clipped importance sampling | input_ids, old_logprobs, advantages | eps_max (6.0) |
dro | Direct reward optimization | input_ids, old_logprobs, advantages | beta (0.05) |
opsd | Hint-conditioned self-distillation | Aligned student and teacher streams; see below | ce_coef (1.0), kl_coef (1.0) |
cross_entropytrains the model to predict target tokens, scaled by per-tokenweights.importance_samplingscales each token's policy-gradient update by its advantage and the current-to-sampling policy probability ratio. It does not clip the ratio, so updates can be noisy when the policies diverge.ppouses the smaller of the unclipped and clipped advantage-weighted ratios, with clipping bounds[1 − clip_low, 1 + clip_high]. This limits the incentive to move token probabilities too far in one update.cispocaps the importance ratio ateps_maxand detaches it from the gradient, while differentiating through the current token log probability. It is used in ScaleRL-style training.droadds a quadratic penalty, controlled bybeta, for moving away from the sampler's log probabilities.
Distillation objectives
For sampled reverse-KL feedback, use an RL loss with a detached per-token
advantage of teacher_logprob - student_logprob. Keep the student sampling
policy's log probabilities in old_logprobs. The
distillation guide explains the estimator and provides
examples for a larger teacher, a self-teacher, and multiple specialists.
cross_entropy also accepts two-dimensional target_tokens and weights per
datum, both shaped [sequence_length, K]. At each prediction position, supply
K candidate token IDs and their teacher probabilities as weights. This trains
against soft targets. Renormalizing over only the top-K tokens changes the
teacher distribution; it is an approximation, not full-vocabulary KL. The
returned per-position log probability is the weight-normalized mean over
candidates, not a vector of their individual log probabilities.
Native self-distillation with opsd
opsd computes full-vocabulary forward KL, KL(teacher || student), from a
hint-conditioned teacher using the same model weights. The teacher stream has
no gradient. It can also include supervised cross-entropy on separate student
positions. It accepts text-only data, not image chunks or an external teacher.
| Stream | Per-datum fields |
|---|---|
| Student | input_ids, target_tokens, loss_mask, kl_mask, kl_ids |
| Teacher | teacher_input_ids, teacher_target_tokens, teacher_kl_mask, teacher_kl_ids |
| Optional attention masks | attention_mask, teacher_attention_mask |
Each stream uses its own sequence length. Targets are next-token IDs aligned
with prediction positions. Binary kl_mask fields select paired positions;
the active kl_ids must be exactly 0..K-1 in each stream and pair identical
target tokens. All masks must be zero at the trailing no-target position.
Student loss_mask and kl_mask must be disjoint.
Unlike the token-summed losses above, opsd optimizes the sum of per-datum
means: each datum contributes ce_coef × CE_sum / N_CE plus
kl_coef × KL_sum / N_KL, with empty components contributing zero. A positive
ce_coef requires SFT tokens in every datum. Set ce_coef=0.0 for KL-only
training; every datum must then contain KL tokens. Coefficients must be finite,
nonnegative, and at least one must be positive.
For a mean over B datums, apply gradient_scale=1 / B once at the optimizer step
on endpoints supporting gradient_scale_v1. Do not also divide by token count.
opsd_objective_sum reports the optimized scalar; opsd_kl and ce_loss report
aggregated token means for the components. See the
aligned example.
Custom losses
The Custom losses guide covers defining a PyTorch objective, preparing it on a model, aligning inputs, and choosing per-example reductions. It includes a decision-model example and explains how River captures and executes the loss on its backend.