Skip to main content
Your coding agent can do everything on this page for you: see the Quickstart. This page says what it sets, and why.
Connecting an agent means changing where its model client sends its requests: a base URL, a key, the model’s name, and two headers that name the workload and the session. Your prompts, your tools and the calls your code makes stay as they are. The Gateway serves two APIs, and your agent keeps the one its calls are written in:
  • Chat Completions: OpenAI’s SDK (chat.completions), any OpenAI-compatible client, or a framework configured with an OpenAI base URL.
  • Messages: Anthropic’s SDK (messages.create, messages.stream), or a framework or agent built on it.

Workloads

A workload is one kind of task your agent repeats, such as support-agent, and the model that serves it. Every request names its workload, and the workload decides which model answers, so that model can change later with no change to your agent. If your product runs different kinds of agent, a support agent and a coding agent, say, each is its own workload, even when they share the code that calls the model. List the org’s workloads and the models a workload can serve, then create one:
A workload’s name is lowercase letters, digits, ., _ and -, starting with a letter or digit, at most 63 characters. It never changes. Start with a large frontier model, generally. The replies it records are what every replay run replays, and the bar candidates are measured against: a scorer that compares each candidate’s reply with the recorded one can’t tell you whether a cheaper model does as well as a strong one would, when the recorded reply is weak.

Titles and labels

When your code creates workloads itself, one per tenant or per generated agent, their names are often generated (agent-3f9c2a1e). Give each a title people can read, and label it with what someone will look for it by: the tenant it serves, its tier, or the team that owns it.
A label is key=value, or a bare key as a tag: --label beta matches any value of beta. Keep labels to values that repeat across workloads, since an id unique to one workload belongs in its name. Retitle a workload, and change its labels when what it serves does:

The Gateway’s base URL and key

ambral workloads get prints everything a client needs to connect to a workload. Use the values it prints:
With --json, they are its connection:

The key

Requests carry an Ambral key, not a model provider’s key such as OpenAI’s or Anthropic’s. Create one on the API keys page of the web app, which shows it once, and store it where your agent reads its secrets: a .env file that git ignores, or your secret store. The CLI can’t create a key, so the secret never passes through a terminal. Don’t paste it to a coding agent, which never needs to see it.

Set up the client

Where your agent builds its client, set: Vercel’s AI SDK provider for Anthropic (@ai-sdk/anthropic) adds only /messages, so its base URL is messagesBaseUrl with /v1 added. A Chat Completions request then reads:
The model can name the workload too, as ambral/auto:support-agent. A request that names it both ways must name the same workload. A request that names a provider’s model is refused with a 400: to serve another model, make it the workload’s primary with ambral workloads set-primary. The Gateway answers in the response format of the API it was called in, with model the model that served the call. It answers errors in that API’s error format too, so your SDK raises its usual error, and the message says what’s wrong: see Errors.

Session ids

A session is each time your agent does its task: a chat with a user, or one pass of an automated workflow, however many model calls it takes. Ambral records, evaluates and routes a session as a whole, so each request has to say which session it belongs to. Send x-session-id on every request: the same id on each request of a session, and a new id for the next session. An id is 8 to 128 characters of A-Z, a-z, 0-9, _ and -, so a UUID fits, and so does an id your app already holds for the conversation. Any other value is ignored. The session id is the easy one to miss. Without it, every request is recorded as a trace of its own, one step long. The Gateway answers such a request with an x-session-id response header, the id of the session it made for it. A client that already names its session another way needs nothing more: the Gateway also reads session_id in the request body, and a header named x-<vendor>-session-id. A session is stored once it has gone quiet for a little while, so a new one takes about half a minute to appear among the workload’s traces.

End users and labels

Two optional headers say more about each session. Send them where your agent sends x-session-id, on every request. Neither ever makes a request fail.

x-ambral-end-user

It names the person the session served, by your own id for them: a customer or account id such as cus_81723, 1 to 128 letters, digits and . _ : @ + -. Send it when your agent serves people, so each person’s sessions can be found later. Prefer an opaque id to a name or an email. Without the header, the Gateway reads the request body: safety_identifier or user (OpenAI’s), then metadata.user_id (Anthropic’s). An agent that already sends one of those needs nothing more. A value that doesn’t fit is ignored. A session has one end user: the first one it was sent. Keep one session id per person’s conversation. A session id reused for another person is still recorded, under the first person.

x-ambral-labels

It labels the session with key=value pairs or bare tags, comma-separated, up to 16:
A key is spelled as a workload’s name is. A value is letters, digits, spaces and . _ : / @ + -, at most 128 characters. A session gathers the labels of every request, and a later value for a key replaces the earlier one. Once a session holds 16 labels, a later request’s new keys are dropped. Label what tells one workload’s sessions apart:
  • The deployment that sent it. Production sends env=prod, a staging deployment env=staging, and a developer trying the agent by hand env=dev. Set the value from your app’s own environment, in the client’s default headers. Then --label env=prod shows only real people’s sessions, to read and to build evals from.
  • A change that ships to some of the traffic. Add a label that names it, such as variant=prompt-v2, and compare its sessions with the others. A model change needs no label: each trace records the model that served it.
Keep labels to values that repeat across sessions. The person served goes in x-ambral-end-user, and the tenant or team a whole workload serves goes in the workload’s own labels. When a label doesn’t fit, the Gateway ignores the whole header. It still serves and records the request, without labels, and says why in a response header: x-ambral-labels: ignored; reason=<reason>, where the reason is invalid_key, invalid_value or too_many. Check a successful response for it once, after you add the header.

Find sessions

ambral traces list lists the org’s sessions, newest first, across every workload. Name a workload to keep only its sessions, and narrow either with flags, each of which must hold:
--label matches a session’s own labels, not its workload’s. To find a tenant’s or team’s sessions, list their workloads with ambral workloads list --label tenant=acme, then each workload’s sessions. A test suite that runs your agent isn’t production traffic, and needs no env label once it reports through one of Ambral’s SDKs: its sessions are recorded as the traces of its offline run, and ambral traces list never lists them. Until a suite is wired that way, label its sessions env=test.

The system prompt

Ambral reads a request’s first message, when it is a system message, as your agent’s prompt: the instructions that stay the same from one session to the next. Everything after it, later system messages included, is the session’s own. Each session’s prompt version, which ambral traces get shows, is a fingerprint of that first message and the tools, the same for every session sent with the same instructions. So your agent’s requests should keep to that layout:
  • The first system message is fixed. It holds the instructions, the same text on every request of the workload, changing only when the instructions do.
  • What changes between sessions comes after it. The customer or account, retrieved documents, today’s date:
    • in Chat Completions, in a later system message, or in the user message;
    • in Messages, at the start of the first user message. Ambral reads system as one system message, however many blocks it has, so keep system to the instructions, in one text block.
A prompt template filled in on each request breaks this: each session gets a prompt version of its own, and a prompt optimized on a run’s export would carry one session’s data into every other. Once the prompt is laid out this way, two sessions of the workload show the same prompt version.

Check the first trace

Run your agent through one short session of a few steps, wait half a minute, then list the workload’s traces:
The newest trace should be your session. If it isn’t there, wait as long again and list once more. Then read the trace whole, with the id the list printed:
Under each reply it shows the model that served it, why it finished, its tokens in and out, and its latency. A call that failed shows its error where it happened. Check that each reply was served by the model you expect, and that no call failed.

When a session has more than one branch

A session has more than one branch when your agent sent a history that differs from what it was answered. A session can look healthy by every other check and still fork at every reply, so find out why before you rely on its traces. ambral traces get prints each branch: look at where each splits off the latest one.
  • Your agent drops the model’s reasoning. This is the usual cause. The branches split at a reply: the same reply, with the same text and the same tool calls, appears twice under the same message, and a session of ten replies has around ten branches. A reply the model reasoned for comes back with that reasoning: in reasoning_details (and in reasoning, when it is text) from Chat Completions, and as thinking and redacted_thinking blocks from Messages. Many frameworks and hand-written loops send the reply back without it, rebuilt from role, content and tool_calls, so it isn’t the reply the model gave. The model then loses its earlier reasoning on every turn, which can lower the quality of multi-step work, and some providers refuse such a request outright. Where your agent appends the model’s reply to the conversation, append the assistant message as it came back, with those fields or blocks and each block’s signature.
  • Your agent rewrites its history on purpose. It retries or regenerates a reply, summarizes or trims older messages when the conversation grows long, or edits a message before sending it again. Each rewrite starts a branch, and that is expected.
  • A call failed, and your agent sent something else next. The failed call is left as a branch of its own, with its error.
  • A voice platform builds the history. Vapi sends back its own version of each reply, so the two copies differ: in their words, when it is its transcriber’s text of the audio, or only in their times and numbers, when it is the reply rewritten for speech. The fix is in the Vapi assistant’s settings. For the transcriber’s text, set modelOutputInMessagesEnabled to true, which works only for a single assistant with an ElevenLabs voice. For the rewriting, set voice.chunkPlan.formatPlan.enabled to false.

When no trace appears

Go through these in order:
  1. The request failed. A 401 means the Ambral key is wrong or isn’t being sent: check that its secret is set where your agent runs. A 402 means the org has no credits available. A 400 that asks for a workload means x-workload is missing, and a 404 that it isn’t the workload’s name as ambral workloads get shows it.
  2. The base URL. It must be the one ambral workloads get prints for the client’s API: baseUrl, /v1 included, for Chat Completions, and messagesBaseUrl for Anthropic’s SDK. Swapping them sends requests to a path the Gateway doesn’t serve.
  3. The model. A request that names a provider’s model in place of ambral/auto is refused with a 400.
  4. The tools. A response header x-ambral-recording: skipped; reason=unsupported_tools means a tool your agent offers can’t be recorded.
When your client fails on a response the Gateway sent, read the Gateway’s contract before guessing what the response holds: https://api.ambral.ai/v1/openapi.json, which needs no key.