Evals

The door the Subcore console runs an agent's eval cases through, one agent turn at a time, with its tools answered from the case. What a project needs for it, its request and stream, which tools may run for real, and how eval turns stay apart from customer conversations.

POST {deployment}/a/{routingKey}/api/evals/turn

From kit 0.44.0, the Subcore console can run an agent's eval cases against the deployment that serves it. The console plays the customer and judges the conversation itself. The deployment answers only the agent's turns, one request each on this door, with the agent's real instructions, model and guards, through the same handler the Playground uses.

What differs from a Playground turn is the tools. A tool the agent calls returns what the case says it returns, and its own code does not run, so an eval needs none of your credentials. A project can let a few read-only tools run for real instead. The exceptions settle as they do in the Playground, so their own code runs: on voice, the lifecycle tools, such as the one that ends a call; on text, the kit's escalation to a person; and core0's handoff from one agent to another. The project's caller-id lookup never runs on an eval turn.

The console makes these requests; you do not. The door is documented here so you can get a project ready for it, decide which tools may run for real, and tell eval traffic apart in a deployment's logs and telemetry.

What your project needs

Nothing calls the door until the console does. Once it does, no tool of yours runs for real until the project lists it, except your lifecycle tools, whose own execute runs on a voice eval turn as it does in the Playground (step 2). For the console to run evals against a deployment:

Build every agent in your createSessionCore0 from config.tools, as the kit already asks, and give none of them core0 skills or mcpServers. The kit answers from the case inside the copies of the tools it hands over there. The factory runs again for every eval turn, and when an agent it builds for that turn can reach another tool, the turn ends with eval_tools_unguarded. So an agent given skills only on voice fails only the voice cases. Another tool is one you built yourself, a lifecycle tool included, a tool from config.tools wrapped again with a new execute (withCondition is fine), the loadSkill tool core0 adds to an agent with skills, or a tool an MCP server registers during the turn. The kit's escalate_to_human and core0's handoff_to_agent never count: each turn hands them to the agent itself, not through your factory.
Keep the execute of your lifecycle tools (transfer, end call and voicemail) free of side effects, because a voice eval runs it.
Optionally, list the read-only tools a case may run for real.
Redeploy on kit 0.44.0.

Authentication

The door takes the console's credential, as every console route on the deployment does: a request the console signs, checked against the deployment's CONSOLE_JWKS, or Authorization: Bearer with its CONSOLE_API_SECRET. A request with neither gets 401, and a deployment with neither configured answers 503. The credential is checked before anything else, the body included.

Only the routing-key path runs a turn. On the bare path, POST /api/evals/turn, a request that passes the credential, body and load checks is refused with 404 and code: "routing_key_required", as every conversation door has been since kit 0.31.0.

The request

POST {deployment}/a/{routingKey}/api/evals/turn
Authorization: Bearer <the deployment's CONSOLE_API_SECRET>
Content-Type: application/json

{
  "sessionId": null,
  "channel": "sms",
  "message": "Hi, can you tell me where my order is?",
  "runId": "run_2026_10_27_nightly",
  "caseId": "order-status-known-customer",
  "callerNumber": "+15555550142",
  "tools": {
    "lookup_customer": { "mode": "mock", "result": { "found": true, "name": "Sam Rivera" } },
    "search_help_center": { "mode": "live" }
  }
}

Field	Type	Description
`sessionId`	string \| null	`null` starts the case's conversation. Every later turn of the case sends the id the first turn's `session` line named: the channel, `-eval-`, then 10 letters, digits, `_` or `-`, such as `sms-eval-V1StGXR8_Z`. A Playground session id is refused
`channel`	string	Required when `sessionId` is `null`: one of the channels the deployment serves, such as `chat`, `sms`, `email` or `voice`. A later turn may leave it out. If it sends one, it must be the session's
`message`	string	Required on every turn, the first included, so the customer always speaks first. The simulated customer's words, not blank and at most 4000 characters
`runId`	string	Required. 1 to 128 letters, digits, `_` or `-`. Every event of the turn carries it, so a run's turns can be read back together
`caseId`	string	Required. 1 to 128 letters, digits, `.`, `_` or `-`. It names the case in the deployment's log line for the turn
`callerNumber`	string	Optional. The caller id the agent sees, at most 32 characters, and a number its lookup tools may be called on. The project's caller-id lookup, `conversation.fetchCustomerFields`, never runs on an eval turn, so a case that needs a customer's details answers the lookup tool instead. Only a lookup tool the project lets run for real takes the number to your systems
`tools`	object	Optional. What each tool returns, by tool id. See how each tool is answered

Send callerNumber and tools again on every turn of a case. A turn can land on an instance of the deployment that never saw the case's first turn, and that instance knows only what the turn in front of it carries.

A voice case runs the way the Playground runs voice: as text, through the voice channel's instructions and tools, without the call policy a relay call runs. It shows what the agent says and which tools it picks, not how a call ends.

How each tool is answered

Each tool on a channel has one mode, which the deployment's describe document reports as the tool's evalMode:

Mode	Which tools	What an eval turn does with it
`intent`	On voice, the project's lifecycle tools (transfer, end call and voicemail). On text, the kit's `escalate_to_human`. And core0's `handoff_to_agent` in handoff mode, unless a tool of yours has that id. On text, a tool with a lifecycle tool's id is an ordinary tool	Settles it as the Playground does, whatever the case says. Its own `execute` runs, because what it returns is the signal the kit or core0 acts on. An answer the request gives for it is ignored
`mock`	Every other tool, unless it is `live`	Returns the case's answer. The tool's own code never runs
`live`	A tool the project lists in `evals.liveToolIds`, on a deployment whose `TELEMETRY_PROFILE` is `full` or unset	Returns the case's answer when the case gives one, and runs for real when the case asks for that

tools maps a tool id to one of two answers:

{
  "lookup_customer": { "mode": "mock", "result": { "found": true, "name": "Sam Rivera" } },
  "search_help_center": { "mode": "live" }
}

{ "mode": "mock", "result": … } hands the model result, which can be any JSON value, as the tool's result. { "mode": "live" } runs the tool, and only a live tool takes it. Asking it of a mock tool refuses the whole turn before it runs, with 400 and code: "eval_live_not_allowed":

{
  "error": "Not allowed to run live here: lookup_customer. The project lists the tools that may (evals.liveToolIds).",
  "code": "eval_live_not_allowed",
  "tools": ["lookup_customer"]
}

tools in the body lists every id the request asked to run live that may only be mocked, including an id no channel wires. A live answer for an intent tool is ignored, not refused. On a deployment whose TELEMETRY_PROFILE is anything but full or unset, the sentence in error says it runs no tool live instead. Do not retry it: the request, or the project's list, has to change first.

A tool the case gives no answer for returns { "error": "no_fixture", "tool": "<tool id>" } to the model, and the stream sends a lifecycle line with "event": "fixture_missing" and the tool's id in data.tool. A request with no tools at all gets that for every tool the agent calls. The console fails a case that reaches one.

The kit's own refusals answer before the case does, whether or not your agents attach onToolCall. A customer lookup on a phone number the conversation never produced is refused, as a real turn refuses it on an agent that attaches the hook, and the case's answer is not used. A tool the agent has switched off on the channel is not in config.tools, so the model never has it.

The case's answers are given inside the copies of the tools the kit hands your createSessionCore0 as config.tools, so a mock tool never runs, whether or not your agents attach onToolCall. If an agent the factory builds can reach a tool that did not come from that hand-over, in any of the ways step 1 lists, the turn ends with an error line whose code is eval_tools_unguarded, before any tool runs. Its message names the tools, loadSkill among them for an agent with skills, and any agent with MCP servers.

The kit's escalate_to_human on text and core0's handoff_to_agent in handoff mode reach the agent with each turn, never through your factory, so no check of the built agents sees them. Both are intents. All handoff_to_agent does is pass the turn to another agent in the same core, so it reaches no system of yours. Describe lists escalate_to_human on the text channels but handoff_to_agent on none, because no channel wires it, so the console shows no row for it.

core0 lets the last tool with an id win, and these two come last. On text, the kit's escalate_to_human replaces a tool of yours by that name, as it does in production, so the case's answer for yours is never read. A tool of yours named handoff_to_agent is yours outside handoff mode, and is answered like any other tool. In handoff mode core0's replaces it, so give yours another name.

tools holds at most 64 entries and 64 KiB of JSON. A tool id is 1 to 128 letters, digits, _, ., : or -.

Letting a tool run for real

A project lists the tools a case may run for real in evals.liveToolIds, in the registerKitRuntime call in agents/core.ts:

registerKitRuntime({
  // …
  evals: { liveToolIds: ['search_help_center'] },
})

List read-only tools only, such as a knowledge-base search, because every run of every case that asks for one repeats the call. The tool worth listing is one whose own behaviour the eval is meant to test: a fixture only shows that the agent reads what it was handed.

The deployment refuses to start when an id names no tool wired to a channel, or names a tool an eval settles as an intent (a voice lifecycle tool, or escalate_to_human on text), which an eval never runs live. A project that sets no evals runs nothing live, which is the default.

A deployment whose TELEMETRY_PROFILE is anything but full or unset runs no tool live, whatever the list says. A live result is real data, and the console's judge would carry it into Subcore. On such a deployment, describe reports every listed tool as mock, and a request that asks live of one gets eval_live_not_allowed. See what structural covers.

The answer

A turn that runs answers 200 with Content-Type: application/x-ndjson and the Playground's stream: one JSON object per line, each with a type. For the request above:

{"type":"session","sessionId":"sms-eval-V1StGXR8_Z","storageKey":"sms-eval-V1StGXR8_Z","channel":"sms","created":true}
{"type":"meta","channel":"sms","simulation":{"label":"…","approximation":true},"deploymentId":"…"}
{"type":"widget","widgetType":"toolCall","id":"tc-4fQe2Lk9","data":{"toolId":"lookup_customer","args":{"phoneNumber":"+15555550142"},"result":{"found":true,"name":"Sam Rivera"},"status":"completed","durationMs":3}}
{"type":"lifecycle","event":"fixture_missing","data":{"tool":"check_order_status"}}
{"type":"widget","widgetType":"toolCall","id":"tc-Zx81mPq0","data":{"toolId":"check_order_status","args":{"orderId":"A-1042"},"result":{"error":"no_fixture","tool":"check_order_status"},"status":"completed","durationMs":1}}
{"type":"token","token":"Thanks, Sam. I can't pull up that order right now, but I can take a message for the team."}
{"type":"finalize"}

`type`	What it carries
`session`	Sent first, at once. `sessionId` is the id to send on the case's next turn, and `storageKey` is the same id. `created` is `true` on the turn that started the session
`meta`	The channel, the Playground's `simulation` note, and `deploymentId`, the build of the deployment that answered. The note has a `label` and `approximation`, which is `true` on every channel but chat
`token`	The reply. Chat sends it a piece at a time. SMS, email and voice send one `token` line with the whole reply, at the end of the turn
`widget`	One per tool call, with `widgetType: "toolCall"` and `data` holding `toolId`, `args`, `result`, `status` and `durationMs`. `args` and `result` are previews cut at 2 KB, marked `argsTruncated` or `resultTruncated` when cut
`lifecycle`	An `event` and its `data`. `fixture_missing` is the one only an eval sends. The rest are the Playground's, such as a transfer settling
`keepalive`	Sent while a turn is quiet, to hold the connection open. Skip it
`finalize`	The last line of a turn that worked
`error`	The last line of a turn that failed partway, with a `message` and a `code`. The code is `eval_tools_unguarded` when an agent could reach a tool the kit did not hand it, in one of the ways step 1 lists, and `execution_error` for any other failure

So a 200 alone does not mean the turn worked. Read to the last line, and treat a stream that ends without finalize or error as a failure. Skip any type you do not know.

A session's turns run one at a time on an instance. Send a case's next turn only after the last line of the one before it.

What it answers

The door checks the credential first, then the body, then its load, then the agent and the session. A request that would fail two of those checks gets the earlier one's answer, and a request refused before the agent check costs the deployment no read of the agent.

Case	Answer
No valid console credential	`401`. A deployment with no console credential configured answers `503`
The body is not a JSON object, or a field is missing or invalid	`400`, with `error` naming the field. A `channel` the deployment does not serve also lists `supportedChannels`
A `channel` that is not the session's	`409`
`live` asked of a `mock` tool	`400` with `code: "eval_live_not_allowed"` and `tools`
The instance already runs 2 eval turns	`429` with `code: "eval_busy"` and `Retry-After: 2`
The bare path, with no routing key	`404` with `code: "routing_key_required"`
A routing key no agent answers on, or an agent the deployment cannot read	The answers a text door gives, set out under test it: `404`, `503` when the deployment could not reach Subcore, or `500` with a `code` naming what failed. A missing required variable is not refused here, as in the Playground
A `sessionId` this agent's door did not start, including one started on another agent's door	`404` with `"error": "Session not found"`
A later turn whose `runId` is not the one its session started under	`409`. A session is one case of one run
The session store could not be read on a later turn	`503`

A refusal's body is { "error": …, "code": … }, or error alone where the table gives no code, and carries no timestamp. Switch on code.

Load

Eval turns run in the process that serves your customers, so one instance runs at most 2 at a time. The next is refused with 429 before the agent is read, so a refused turn costs nothing:

{ "error": "This instance is running as many eval turns as it allows", "code": "eval_busy" }

Retry it after the Retry-After seconds. Customer and Playground turns never count against the limit and never wait on it. It applies per instance, so a deployment running several instances takes more eval turns at once. A turn holds its place for at most 3 minutes, so a turn that never settles cannot keep the door shut. Past that, its place is given to the next turn even if it is still running, so an instance with stuck turns can briefly run more than 2.

Kept apart from your conversations

Every eval turn is recorded under the agent's slug, with run_type: "eval" and the request's runId as its run_id, at the deployment's own telemetry profile. That keeps it out of production numbers, and GET /telemetry/runs/{runId} reads one run's events back. On a structural deployment those events leave without conversation content, like every other event it sends.

An eval session never reaches the project's conversationRecords.onClosed hook: no record closes for it, however long it stays quiet. It keeps no phone number a takeover could text, and its ids are not the Playground's, so neither door accepts the other's.

Everything a case sends passes through Subcore, and so does each reply the agent gives, as in the Playground. Write cases with test data only.

What the deployment announces

The manifest at /.well-known/subcore-agent.json gains a capability and an endpoint:

{
  "capabilities": { "evals": { "version": 1 } },
  "endpoints": {
    "evalTurn": {
      "method": "POST",
      "path": "/api/evals/turn",
      "description": "Eval: one agent turn of a case the console runs, tools answered from the case (NDJSON stream)"
    }
  }
}

The path is served under each agent's {tenantPath}. The door is not under capabilities.channelEndpoints, so the MCP server builds no conversation tool from it.

The describe document gains capabilities.evals: true, and every tool in each channel's list under tools carries its evalMode on that channel:

"tools": [
  {
    "channel": "sms",
    "tools": [
      { "id": "lookup_customer", "description": "…", "evalMode": "mock" },
      { "id": "search_help_center", "description": "…", "evalMode": "live" },
      { "id": "escalate_to_human", "description": "…", "locked": true, "outcome": "escalated", "evalMode": "intent" }
    ]
  }
]

Both are additive, so describeVersion stays 1. A deployment on a kit before 0.44.0 announces neither. The console offers each tool what its evalMode allows, and describe_deployment returns both fields.

On this page