Learning from examples

Supervised fine-tuning (SFT) trains an LLM to produce responses like the examples you provide. You choose the demonstrations and which parts of them contribute to learning. River computes the training updates.

By the end of this chapter, you should be able to explain what the loss measures, why we mask prompts, and why a lower training loss does not prove a better model. For executable code, follow Your first SFT run.

Begin with the behavior

Suppose your application needs to turn a support request into a structured record. A demonstration pairs what the model will see with what it should say:

The response is the supervision. It might be a JSON record, a paragraph, a tool call, or a longer conversation. Training encourages the model to produce those responses in their corresponding contexts.

Write the task definition before collecting examples. Decide what a correct response contains, what information is available at inference time, and how you will score unseen cases. A model can imitate inconsistent examples without learning the behavior you intended.

What the model is learning

A model produces text as a sequence of tokens: pieces of text represented by integer IDs. Given the context so far, it assigns probabilities to possible next tokens. SFT uses those probabilities to measure how well the model predicts your demonstrations and to compute the training objective.

During training, the model sees the correct prefix from the demonstration and is asked to predict the next demonstrated token. This is called teacher forcing. During generation, it instead continues from the tokens it has already produced, including any mistakes.

The cross-entropy loss measures how little probability the model assigned to the desired token. For one prediction, it is the negative natural logarithm of that probability:

If the desired token has probability 0.1, its loss is about 2.30. At probability 0.5, the loss is about 0.69. At probability 0.9, it is about 0.11. A smaller loss means the model assigned more probability to that token; it is not a percentage of correctly answered tasks.

Learn from the response, condition on the prompt

The prompt provides context. Usually, you want the model to learn the response without spending the training objective on reproducing the prompt. A loss mask assigns zero weight to prompt predictions and nonzero weight to response predictions. The prompt still influences every response prediction.

The important alignment is that input position i predicts token i + 1. The last prompt position therefore predicts the first response token.

River's core training data uses input_ids, optional target_tokens, and per-position weights. The SFT walkthrough builds these arrays explicitly so you can inspect them. For conversations, use the chosen model's chat format and include the expected end-of-response markers. Training and inference must agree on that format.

What happens in one step

A batch is a collection of training examples. A forward/backward operation computes the loss and accumulates gradients: the direction in which the trainable weights should change. An optimizer step applies an update using those gradients. The learning rate controls the scale of that update.

In River, model.forward_backward(..., loss_fn="cross_entropy") and model.optim_step(lr=...) expose those two operations. model.train_step(...) combines them for the common case. This is an interface to training; you do not implement differentiation or move tensors between GPUs.

River's core loss is a weighted sum over token positions. Doubling the number of contributing tokens can change both the reported loss and gradient scale. Compare runs with consistent weighting and normalization; a larger raw loss can simply come from a larger batch. See Loss functions.

The examples train LoRA adapters, a smaller set of trainable parameters added to a fixed base model. Think of an adapter as the learned adjustment you save alongside the base model's identity. Choosing an adapter explains the settings you control.

Build a dataset that can teach the task

Start with a small collection you can inspect. Include realistic inputs, consistent answers, and cases where the correct behavior is to decline, request missing information, or use a tool. Remove accidental secrets and information that the model would not have when serving a real user.

Split the data before tuning. Keep related records, duplicate questions, and variants of the same underlying case together so that the held-out set tests new cases. A random row split can overstate generalization when examples are near duplicates.

Sample from the base model first. If it already performs well, a prompt change may be sufficient. If training is warranted, keep that baseline for comparison.

Decide whether training helped

Evaluate with the prompt format and generation settings you intend to use. Inspect actual responses as well as aggregate scores. Save checkpoints so you can choose the model that performs best on held-out tasks, rather than assuming the last update is best.

Try it: after running the SFT example, compare a training input, an unseen input of the same kind, and one outside the training distribution. Explain what each result does—and does not—tell you about generalization.