Deploy and serve

Dedicated streaming deployments

Use a dedicated deployment to serve your trained model to applications and agents. River reserves serving capacity for a saved checkpoint and exposes an OpenAI-compatible endpoint, including streaming responses. Training sessions and production deployments have separate lifecycles: ending a training session does not stop a deployment. Use a team API key with deployment access for the checkpoint's base model; contact River to enable access. Personal API keys cannot create deployments.

pip install river-client openai

Save the trained model with model.save_weights("serving-v1", mode="inference"), then use the returned checkpoint path:

import os

import river_client as river
from openai import OpenAI

api_key = os.environ["RIVER_API_KEY"]
client = river.Client(api_key=api_key, endpoint="api.river.ai")
deployment = client.create_deployment(
    checkpoint="river://YOUR_RUN_ID/sampler_weights/serving-v1",
    unified_replicas=1,
    idempotency_key="serving-v1",
    wait=True,
)
print("Deployment:", deployment.id, deployment.base_url)

with OpenAI(api_key=api_key, base_url=deployment.base_url) as inference:
    with inference.chat.completions.create(
        model=deployment.model,
        messages=[{"role": "user", "content": "Hello"}],
        stream=True,
    ) as stream:
        for chunk in stream:
            if chunk.choices:
                print(chunk.choices[0].delta.content or "", end="", flush=True)

The URL selects the checkpoint. It already includes the OpenAI API prefix; pass it unchanged as base_url. Creation is asynchronous: wait=True waits for serving capacity. To require every requested replica to be ready, call client.wait_for_deployment(deployment.id, until=("ready",)). Reuse an idempotency key when retrying the same creation request; use a new key for a new deployment.

Capacity is specified in replicas. River selects the GPU shape for the model; one replica does not necessarily mean one GPU. Choose either unified_replicas or both prefill_replicas and decode_replicas at creation. The topology cannot be changed afterwards.

Scale, stop, and delete

client.list_deployments()
client.scale_on_target(deployment.id, unified_replicas=2)
client.get_deployment_usage(deployment.id)

# Release serving capacity while keeping the URL and checkpoint.
client.scale_on_target(deployment.id, unified_replicas=0)
client.wait_for_deployment(deployment.id, until=("scaled_to_zero",))

# Resume on the same URL.
client.scale_on_target(deployment.id, unified_replicas=1)
client.wait_for_deployment(deployment.id)

# Release the deployment when it is no longer needed.
client.delete_deployment(deployment.id, wait=True)

Scaling sets absolute targets, not increments. For a prefill/decode deployment, scale both roles to zero together and both back up to resume. Scale-down and deletion drain active work before releasing resources. Requested GPU-hours start at create/scale acceptance and stop at scale-to-zero/delete acceptance; they include provisioning and outage time. Usage records report observed allocation separately.

Use a fresh checkpoint name and a new deployment to update weights while the old deployment continues serving. Saving new weights at an existing checkpoint path does not reload running workers. To update at the same URL, scale to zero, wait for draining to finish, save the new weights, then restore capacity.

Deployments also support stateless /responses requests: set store=False and send conversation history on each request. Stored responses, previous_response_id, and background processing are unsupported. If a stream fails after it starts, the application must decide whether to start a new request; an interrupted stream cannot be resumed.

See the Python API reference for the deployment methods and parameters.