Eval runs

A run plays an agent's eval cases against it, one case at a time. For each case the Subcore console plays the customer, sends each agent turn to the agent's deployment on its eval door, and judges the conversation at the end. These routes start the same runs the console's Run button starts, and read every run of the agent, whoever started it.

Every route needs the manage-agents permission, counts against the agent's rate budget, and answers 404 NOT_FOUND for an agent that is not the organization's or is retired. Starting and stopping a run also need the run-evals permission. The MCP server's eval tools call the same handlers.

What a run costs

A run is not free, and it is not quick:

• It spends the agent's deployment. Every agent turn is a request to it, with the agent's real model, on your deployment's model key.
• It spends Subcore's models. The simulated customer and the judge run on Subcore's side.
• It takes minutes. Cases run one at a time, each up to its maxTurns agent turns, and a deployment answers at most two eval turns at once, across every run on it. A suite of 20 cases is several minutes.

One run per agent goes at a time. A second start answers 409 RUN_IN_PROGRESS until the first one ends. Tools return what each case says they return, and only the tools your project lists may run for real, so a run does not reach your systems unless you let it. See evals.

Start a run

POST /agents/{slug}/evals/runs

The body holds exactly one of:

curl -X POST -H "Authorization: Bearer $SUBCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: seed-2026-10-30-reschedules" \
  "https://api.subcore.ai/agents/front-desk/evals/runs" \
  -d '{ "suite": "reschedules" }'

{ "replayed": false, "run": { "runId": "run_4f0c…", "status": "running", "total": 12, … } }

It answers 201 once the run has started, with the cases fixed as they were: an edit, an archive or a delete after this point applies to the next run. Both fields, neither, or another field is 400 INVALID_BODY, and a body over 4 KiB is 413 PAYLOAD_TOO_LARGE.

Before it starts anything, the console checks the run can go. It refuses:

Retrying safely: the Idempotency-Key

A start that timed out may have started a run. Send an Idempotency-Key header, any 1 to 255 characters you choose, and retry with the same key:

• A key that already started a run, with the same agent and body from the same API key, answers 200 with replayed: true and that run, whatever its state now: still running, complete, or since edited around. Nothing new starts.
• A key that started something else, another agent, another body or through another API key, answers 409 IDEMPOTENCY_KEY_REUSED. Keys share one space across your organization, so make them specific, such as a date and a suite.
• A start refused for any reason in the table above stores nothing, so the same key works once the cause is fixed, such as a busy agent being free again.
• A key outside 1 to 255 characters is 400 INVALID_IDEMPOTENCY_KEY.

Two rarer answers come after the run was written:

A key lives as long as the run it started. Without a key, a retry after a timeout can start a second run once the first ends.

Starts have a budget of their own, beside the agents API's: 10 a minute for each agent, per key, shared by this route and the MCP server's start_eval_run. It is taken once a request has passed its own checks, so a malformed body costs nothing, and a replay counts as a start. Over it is 429 RATE_LIMITED with scope: "eval-run-start" and Retry-After.

Stop a run

POST /agents/{slug}/evals/runs/{runId}/stop

curl -X POST -H "Authorization: Bearer $SUBCORE_API_KEY" \
  "https://api.subcore.ai/agents/front-desk/evals/runs/run_4f0c…/stop"

It answers 200 with { "run" }, its stopRequestedAt set. The run finishes the case in flight, so its status stays running for up to one more case, then completes with endReason: "stopped". Asking again answers the same, also once it has completed that way.

The run object

{
  "runId": "run_4f0c…",
  "suite": "reschedules",
  "source": "api",
  "status": "running",
  "endReason": null,
  "stopRequestedAt": null,
  "total": 12,
  "resultsWritten": 5,
  "passed": null,
  "failed": null,
  "currentCase": { "caseId": "reschedule-weekend", "sid": "sms-eval-…" },
  "startedAt": "2026-10-30T09:20:01.554Z",
  "lastActiveAt": "2026-10-30T09:23:40.012Z",
  "finishedAt": null,
  "deploymentIdentity": { "deploymentIds": ["…"], "configDigest": "…" }
}

A run fails, with endReason: "failed", when the agent's configuration or the deployment it is reached through changes during it, or when the runner itself fails. A case cut off that way records why in its reasoning.

List the runs

GET /agents/{slug}/evals/runs

Every run of the agent, newest first: those started here, in the console, and local runs that reported in.

Read a run

GET /agents/{slug}/evals/runs/{runId}

{ "run": Run }, or 404 RUN_NOT_FOUND. runId may be any 1 to 255 characters, as a local run's id can be.

Read the results

GET /agents/{slug}/evals/runs/{runId}/results

{
  "run": { "runId": "run_4f0c…", "status": "complete", "passed": 10, "failed": 2, … },
  "items": [
    {
      "caseId": "reschedule-delivery",
      "passed": false,
      "reasoning": "Gated: expected tool calls — lookup_customer was never called",
      "sid": "sms-eval-…",
      "mode": null,
      "createdAt": "2026-10-30T09:21:12.004Z"
    }
  ],
  "nextCursor": null
}

The run comes with every page, so one call polls its status and reads its verdicts. Results are oldest first, 25 a page by default and at most 100, and without their details.

Polling

Poll GET …/runs/{runId}/results, from the first page each time, until the run's status is complete, then page through with nextCursor until it is null to read every verdict. Every 10 seconds is plenty: each case is a whole conversation.

Read one result

GET /agents/{slug}/evals/runs/{runId}/result?caseId={caseId}

{ "result" }: one verdict with details. For a run the console ran, details holds the case as it ran, the models that played the customer and judged, the agent's tool calls in order, and the judge's checklist. For a local run, it is whatever the harness reported. The case id is a query parameter because a local run's can be any text, up to 255 characters. Without it, or past that, the answer is 400 VALIDATION_ERROR, and a run with no result for it is 404 CASE_NOT_FOUND.

Errors

The codes above, plus the agents API's own on errors. 403 FORBIDDEN on a start or stop means the key lacks the run-evals permission.