Multi-teacher distillation

Multi-teacher training need not average several teachers' logits at every token. Start with a dataset carrying an explicit domain label and route each task to one specialist. This is the routing pattern studied in MOPD. Separate teachers can share an architecture and initial checkpoint; they do not have to be larger than the student.

Load and route to specialists

Load validated specialist checkpoints as live River model handles using session.create_model(base_model=..., lora=..., checkpoint=...), with each checkpoint's matching base model and adapter configuration. Build teachers, a dictionary from dataset domain to handle. All teachers must satisfy the same tokenization contract as the student. task_batch contains your domain and question fields. Use distill_step from the basic walkthrough and the prompt_ids helper with the student's tokenizer tok.

records = []
for task in task_batch:
    ids = prompt_ids(task["question"])
    teacher = teachers[task["domain"]]
    records.append((ids, teacher, ids))

result = distill_step(student, records)

Preserve the task mixture

Build a fresh mixed batch on every iteration. Because distill_step gives each response equal weight, the proportions of tasks in the batch set the domain weights.

Record teacher checkpoint IDs with the rollouts, and measure held-out success separately for every domain.

Choose the teacher combination deliberately

Routing, averaging teacher probabilities, and averaging teacher log probabilities define different targets. A probability mixture permits alternatives favored by different teachers; a weighted sum of reverse KLs corresponds to a normalized geometric combination at a fixed prefix. Neither is automatically an improvement over a good specialist. Begin with explicit domain routing before introducing a learned router or an ensemble objective.

Extend the recipe to agents

For agents, apply the same idea to each model-generated span with its actual conversation prefix, including preceding tool results. Tool outputs supply context and receive no direct distillation loss. Keep teacher-only hints out of the student's context, and score the student's recorded actions rather than substituting a teacher's independently generated tool trajectory.