Eval cases

An eval case is a scripted conversation the Subcore console runs against an agent: a simulated customer opens with a message and pursues a goal, the agent answers through its deployment's eval door, and a judge scores the conversation against the case's criteria. Every tool the agent calls returns what the case says it returns. These routes write the same cases the console's Evaluations page shows, so a case written here can be run from the console, and the other way round. Eval runs starts them.

Every route needs the manage-agents permission, counts against the agent's rate budget, and answers 404 NOT_FOUND for an agent that is not the organization's or is retired. The MCP server's eval tools call the same handlers.

{caseId} is the case's id, the slug field below. The writes are checked and stored by the Subcore console, which holds the case rules, so a case these routes accept is one the console's own editor accepts, and the other way round.

The case object

{
  "suite": "reschedules",
  "slug": "reschedule-delivery",
  "channel": "sms",
  "input": "Hi, I want to change my delivery date",
  "goal": "Move the delivery to next Tuesday",
  "persona": "Brief, a little impatient",
  "callerNumber": "+15555550142",
  "maxTurns": 8,
  "successCriteria": ["Confirms the new date back to the customer"],
  "failureConditions": ["Promises a date the lookup did not offer"],
  "expectedToolCalls": ["lookup_customer"],
  "tools": {
    "lookup_customer": { "mode": "mock", "result": { "found": true, "name": "Sam" } },
    "search_help_center": { "mode": "live" }
  },
  "archivedAt": null,
  "createdAt": "2026-10-30T09:12:44.120Z",
  "updatedAt": "2026-10-30T09:12:44.120Z"
}

This is the shape the console's case sheet shows in its JSON view, and the shape every console result keeps as the case it ran. A case copied from either can be sent here as it is, and a case read here can be sent back unchanged: the three read-only fields are removed before it is checked.

Tool answers

Each entry in tools answers one tool the agent may call during the case:

A tool the case does not answer returns that it has no answer, and the case fails, naming it. At most 64 tools, and at most 64 KiB of answers as JSON. A tool id is 1 to 128 of A-Z a-z 0-9 _ . : -. A tool that ends or hands off the conversation, which describe marks evalMode: "intent", settles on its own and cannot be answered: a case that tries is refused. From kit 0.45.0 a tool may publish the shape of what it returns as result in describe, which is the shape a mock result should take.

Rules a case is held to

A case is refused, and nothing is written, when:

• a field is not one of those above. A misspelt key is refused rather than dropped, so a field is never reset without a word;
• a text field is not a string, or persona or callerNumber is neither a string nor null, or maxTurns is not a number;
• a value is past its limit above, or channel, input or goal is missing;
• channel is constructor, or a tool id under tools is __proto__: names JavaScript reserves, which a case cannot use;
• a criterion holds a line break: give each its own line;
• it answers a tool the deployment marks intent;
• its id is new, or changed, and made only of dots, such as . or .., which no URL can carry. A case already stored under one keeps it.

The refusal is 400 INVALID_EVAL_CASE and names the field, such as { "code": "INVALID_EVAL_CASE", "field": "tools.transfer_call", … }.

A suite holds at most 50 cases, and a run runs at most 50. A case that would be its suite's 51st is refused with 409 SUITE_FULL, which names no field. Archived cases count, so delete one to make room.

Some rules depend on the agent's deployment, which can change after a case is saved. Those do not refuse a save. The first one the case breaks comes back as warning:

The deployment's rules are read from the copy of its description the console keeps, as the console's editor reads them.

The id lock

A case's id is fixed once the case has a result, because its results are filed under it, and while a run that is still going holds it. A rename then answers 409 CASE_ID_LOCKED with reason has_results or in_running_run. Every other field can still change.

List the suites

GET /agents/{slug}/evals/suites

curl -H "Authorization: Bearer $SUBCORE_API_KEY" \
  "https://api.subcore.ai/agents/front-desk/evals/suites"

{
  "items": [{ "suite": "reschedules", "liveCases": 12, "archivedCases": 3 }],
  "nextCursor": null
}

A suite exists while it holds a case, archived ones included. There is nothing to create: a case's suite makes one. By name, 25 a page by default and at most 100.

List the cases

GET /agents/{slug}/evals/cases

curl -H "Authorization: Bearer $SUBCORE_API_KEY" \
  "https://api.subcore.ai/agents/front-desk/evals/cases?suite=reschedules"

{ "items": [Case], "nextCursor" }, by suite and then id.

Read one case

GET /agents/{slug}/evals/cases/{caseId}

{ "case": Case }, archived or not. 404 CASE_NOT_FOUND when the agent has no case with that id.

Create or replace a case

PUT /agents/{slug}/evals/cases/{caseId}

curl -X PUT -H "Authorization: Bearer $SUBCORE_API_KEY" \
  -H "Content-Type: application/json" \
  "https://api.subcore.ai/agents/front-desk/evals/cases/reschedule-delivery" \
  -d '{
    "suite": "reschedules",
    "channel": "sms",
    "input": "Hi, I want to change my delivery date",
    "goal": "Move the delivery to next Tuesday",
    "successCriteria": ["Confirms the new date back to the customer"],
    "expectedToolCalls": ["lookup_customer"],
    "tools": { "lookup_customer": { "mode": "mock", "result": { "found": true } } }
  }'

{ "created": true, "case": { "slug": "reschedule-delivery", "suite": "reschedules", … }, "warning": null }

• With no case under {caseId}, it creates one and answers 201. With one, it replaces every field of it and answers 200. So the same request run twice leaves the same case, and a seeding script can send it on every run.
• It is a whole replace. A field you leave out takes its default: suite console, persona and callerNumber null, the three lists empty, tools {} and maxTurns 8. Read the case first and send it back with your change.
• slug may be left out; it is taken from the path. A slug that differs from {caseId} renames the case under {caseId}, unless its id is locked. Renaming onto an id the agent already has is 409 CASE_ID_TAKEN. Renaming a case that does not exist is 400 CASE_ID_MISMATCH.
• warning is null, or { "field", "message" } naming the first deployment rule the case breaks. It never stops the save.
• A body over 768 KiB is 413 PAYLOAD_TOO_LARGE, one that is not JSON 400 INVALID_JSON, and one that is not an object 400 INVALID_BODY.

Archive and unarchive

POST /agents/{slug}/evals/cases/{caseId}/archive
POST /agents/{slug}/evals/cases/{caseId}/unarchive

An archived case keeps its history and its place in its suite's 50, and is left out of later runs. Both answer 200 with { "case" }. Archiving a case that is already archived keeps its archivedAt, so either call can be repeated.

Delete a case

DELETE /agents/{slug}/evals/cases/{caseId}

204, and 404 CASE_NOT_FOUND once it is gone. Its results stay with the runs that judged it, under its id, as they do when it is deleted in the console.

Errors

The rest, such as 401, 403 and 429, are the agents API's own, on errors.