Skip to content
Training objectives

Loss functions

Choose a built-in objective below, or write a custom PyTorch loss for RL, distillation, or supervised learning. Custom losses run on River's training backend and have their own guide, with examples and AsyncTrainer integration.

Built-in losses

Pass the loss name as loss_fn to forward_backward, with loss-specific keyword arguments, for example: model.forward_backward(data, loss_fn="cispo", eps_max=8.0).

Cross-entropy and the four RL objectives below sum token losses. To exclude a token, set its weights value to 0.0 for cross_entropy, or its advantages value to 0.0 for an RL loss. Mask prompt tokens this way to train only on the completion. opsd uses a different reduction, described below.

loss_fnUseRequired per-datum fieldsArguments (default)
cross_entropySupervised fine-tuninginput_ids, weights (target_tokens optional)—
importance_samplingOff-policy policy gradientinput_ids, old_logprobs, advantages—
ppoClipped policy optimizationinput_ids, old_logprobs, advantagesclip_low (0.2), clip_high (0.2)
cispoClipped importance samplinginput_ids, old_logprobs, advantageseps_max (6.0)
droDirect reward optimizationinput_ids, old_logprobs, advantagesbeta (0.05)
opsdHint-conditioned self-distillationAligned student and teacher streams; see belowce_coef (1.0), kl_coef (1.0)
  • cross_entropy trains the model to predict target tokens, scaled by per-token weights.
  • importance_sampling scales each token's policy-gradient update by its advantage and the current-to-sampling policy probability ratio. It does not clip the ratio, so updates can be noisy when the policies diverge.
  • ppo uses the smaller of the unclipped and clipped advantage-weighted ratios, with clipping bounds [1 − clip_low, 1 + clip_high]. This limits the incentive to move token probabilities too far in one update.
  • cispo caps the importance ratio at eps_max and detaches it from the gradient, while differentiating through the current token log probability. It is used in ScaleRL-style training.
  • dro adds a quadratic penalty, controlled by beta, for moving away from the sampler's log probabilities.

Distillation objectives

For sampled reverse-KL feedback, use an RL loss with a detached per-token advantage of teacher_logprob - student_logprob. Keep the student sampling policy's log probabilities in old_logprobs. The distillation guide explains the estimator and provides examples for a larger teacher, a self-teacher, and multiple specialists.

cross_entropy also accepts two-dimensional target_tokens and weights per datum, both shaped [sequence_length, K]. At each prediction position, supply K candidate token IDs and their teacher probabilities as weights. This trains against soft targets. Renormalizing over only the top-K tokens changes the teacher distribution; it is an approximation, not full-vocabulary KL. The returned per-position log probability is the weight-normalized mean over candidates, not a vector of their individual log probabilities.

Native self-distillation with opsd

opsd computes full-vocabulary forward KL, KL(teacher || student), from a hint-conditioned teacher using the same model weights. The teacher stream has no gradient. It can also include supervised cross-entropy on separate student positions. It accepts text-only data, not image chunks or an external teacher.

StreamPer-datum fields
Studentinput_ids, target_tokens, loss_mask, kl_mask, kl_ids
Teacherteacher_input_ids, teacher_target_tokens, teacher_kl_mask, teacher_kl_ids
Optional attention masksattention_mask, teacher_attention_mask

Each stream uses its own sequence length. Targets are next-token IDs aligned with prediction positions. Binary kl_mask fields select paired positions; the active kl_ids must be exactly 0..K-1 in each stream and pair identical target tokens. All masks must be zero at the trailing no-target position. Student loss_mask and kl_mask must be disjoint.

Unlike the token-summed losses above, opsd optimizes the sum of per-datum means: each datum contributes ce_coef × CE_sum / N_CE plus kl_coef × KL_sum / N_KL, with empty components contributing zero. A positive ce_coef requires SFT tokens in every datum. Set ce_coef=0.0 for KL-only training; every datum must then contain KL tokens. Coefficients must be finite, nonnegative, and at least one must be positive.

For a mean over B datums, apply gradient_scale=1 / B once at the optimizer step on endpoints supporting gradient_scale_v1. Do not also divide by token count. opsd_objective_sum reports the optimized scalar; opsd_kl and ce_loss report aggregated token means for the components. See the aligned example.

Custom losses

The Custom losses guide covers defining a PyTorch objective, preparing it on a model, aligning inputs, and choosing per-example reductions. It includes a decision-model example and explains how River captures and executes the loss on its backend.