Test Suites API

Test suites let you attach automated, layered checks to a prompt, routing policy, or MCP tool and run them on demand or automatically when the target changes. Each suite groups test cases across four layers — prompt quality, routing, governance, and MCP integration — and records a run history you can gate publishing on.

All endpoints require an authenticated session and operate within your company scope; requests without a valid session return 401 Unauthorized. Requests that reference a suite outside your company receive 404 Not Found (existence is not leaked across tenants).

Endpoints

Method	Path	Description
GET	/api/v1/test-suites	List the company's test suites
POST	/api/v1/test-suites	Create a test suite
GET	/api/v1/test-suites/[id]	Fetch one test suite
PATCH	/api/v1/test-suites/[id]	Update a test suite
DELETE	/api/v1/test-suites/[id]	Delete a test suite
GET	/api/v1/test-suites/[id]/cases	List cases in a suite
POST	/api/v1/test-suites/[id]/cases	Add a case to a suite
PATCH	/api/v1/test-suites/[id]/cases	Update a case (caseId in body)
DELETE	/api/v1/test-suites/[id]/cases?caseId=...	Delete a case
POST	/api/v1/test-suites/[id]/run	Trigger a run
GET	/api/v1/test-suites/[id]/run	List run history

Layer values used throughout: PROMPT_QUALITY, ROUTING, GOVERNANCE, MCP_INTEGRATION.

Suites

List suites

GET /api/v1/test-suites

Response 200

{
  "data": [
    {
      "id": "clxxx…",
      "name": "Checkout prompt regression",
      "description": "Guards the checkout assistant prompt",
      "targetType": "STORED_PROMPT",
      "targetId": "clprompt…",
      "enabledLayers": ["PROMPT_QUALITY", "GOVERNANCE"],
      "runOnVersionChange": true,
      "runOnPublish": false,
      "blockPublishOnFail": true,
      "createdBy": "cluser…",
      "createdAt": "2026-08-29T10:00:00.000Z",
      "updatedAt": "2026-08-29T10:00:00.000Z",
      "testCaseCount": 12,
      "testRunCount": 3,
      "lastRun": { "id": "clrun…", "status": "PASSED", "createdAt": "…" }
    }
  ],
  "meta": { "total": 1 }
}

Create a suite

POST /api/v1/test-suites
Content-Type: application/json

Body

Field	Type	Required	Notes
name	string	yes	1–200 chars
targetType	enum	yes	STORED_PROMPT, ROUTING_POLICY, or MCP_TOOL
targetId	string	yes	ID of the target the suite tests
description	string	no	≤ 1000 chars
enabledLayers	string[]	no	Subset of the four layer values

Returns 201 with the created suite, 400 for an invalid body, 401 if unauthenticated, 404 if the target doesn't exist or you can't see it, and 409 if a suite with the same name already exists for the target.

Which targets you can use: for STORED_PROMPT and MCP_TOOL, targetId (the prompt's ID or its human-readable promptId) must be a prompt you can see: your own, a PUBLIC one, or one SHARED with you or your team. Account owners and admins can use any prompt in the company. A colleague's PRIVATE prompt returns the same 404 as a prompt that doesn't exist. Routing policies only need to belong to your company.

Example: Ben has a PRIVATE prompt PR-salary-review. Ana creating a suite with "targetId": "PR-salary-review" gets 404. Once Ben shares the prompt with Ana, the same request returns 201.

Fetch, update, or delete a suite

GET    /api/v1/test-suites/[id]
PATCH  /api/v1/test-suites/[id]
DELETE /api/v1/test-suites/[id]

PATCH accepts any subset of name, description, enabledLayers, and the automation flags (runOnVersionChange, runOnPublish, blockPublishOnFail). A 409 Conflict is returned if the update collides with an existing suite constraint. Unknown IDs return 404.

Cases

List cases

GET /api/v1/test-suites/[id]/cases

Response 200

{
  "data": [ { "id": "clcase…", "name": "…", "layer": "ROUTING", "isActive": true } ],
  "meta": { "suiteId": "clxxx…", "suiteName": "Checkout prompt regression", "total": 8, "activeCount": 7 }
}

Add a case

POST /api/v1/test-suites/[id]/cases
Content-Type: application/json

Body

Field	Type	Required	Notes
name	string	yes	1–200 chars
layer	enum	yes	One of the four layer values
input	object	no	Layer-specific inputs (default {})
expectations	object	no	Layer-specific assertions (default {})
description	string	no	≤ 1000 chars
order	integer	no	Display order (default 0)
isActive	boolean	no	Default true
timeoutMs	integer	no	1000–300000, default 30000

PROMPT_QUALITY cases on a stored prompt

When the suite's target is a stored prompt, a PROMPT_QUALITY case runs that prompt: its latest production (PROD) version, with the case's variables filled into its and the prompt's own temperature, output limit (maxTokens) and reasoning dial. A case's own temperature or maxTokens overrides them. input.prompt is not used for this target type. The prompt is resolved as the suite's creator, so a suite pointed at a prompt its creator can't see (another member's PRIVATE prompt) fails with Stored prompt not found.

input field	Effect
variables	Values for the prompt's
versionId	Run a specific version of this prompt instead of the latest PROD
systemPrompt	Replace the version's system prompt for this case
temperature, maxTokens, model, policyId	Execution overrides

If the prompt has no PROD version, versionId isn't one of its versions, the prompt is encrypted, or the suite's creator belongs to a different home company than the prompt, the case fails with a storedPromptContent assertion that names the reason. It never passes without running the prompt.

Example: a suite on the prompt "Summarizer" (Summarize for a manager: ) with this case:

{
  "name": "Summary stays short",
  "layer": "PROMPT_QUALITY",
  "input": { "variables": { "text": "Q3 incident report: ..." } },
  "expectations": { "minLength": 20, "maxLength": 600 }
}

The run sends Summarize for a manager: Q3 incident report: ... with the version's system prompt, then checks the answer's length.

Update or delete a case

PATCH /api/v1/test-suites/[id]/cases takes the same fields plus a required caseId in the body. DELETE /api/v1/test-suites/[id]/cases?caseId=... takes caseId as a query parameter and returns 400 if it is missing.

Runs

Trigger a run

POST /api/v1/test-suites/[id]/run
Content-Type: application/json

Body

Field	Type	Required	Notes
trigger	enum	no	MANUAL (default), VERSION_CHANGE, PRE_PUBLISH, or SCHEDULED

Response 202 Accepted — the run is queued and executes asynchronously.

{
  "data": {
    "id": "clrun…",
    "suiteId": "clxxx…",
    "trigger": "MANUAL",
    "triggeredBy": "cluser…",
    "status": "QUEUED",
    "totalCases": 12,
    "createdAt": "2026-08-29T10:05:00.000Z"
  },
  "message": "Test run started"
}

A suite with no active cases returns 400; a run already in progress for the suite returns 409 Conflict. A suite whose target prompt is deactivated also returns 409 Conflict ("The prompt this suite tests is deactivated. Reactivate it to run the suite."), and no run is created. So does a suite whose target prompt was deleted, is no longer visible to you, or (for an MCP_TOOL suite) is no longer published as an MCP tool: "The prompt this suite tests no longer exists, or you no longer have access to it." (for MCP_TOOL: "The MCP tool this suite tests no longer exists, is no longer published, or you no longer have access to it."). 404 is returned only when the suite itself doesn't exist in your company. In both 409 cases you can still edit the suite and its cases. If the prompt is deactivated while a run is in progress, the remaining cases fail with "Stored prompt … is deactivated" instead of calling a model.

List run history

GET /api/v1/test-suites/[id]/run

Returns the suite's past runs (most recent first) with their status and per-layer results.