Synthetic Testing API

Synthetic testing enables automated, scheduled testing of AI providers and models against curated prompts to monitor performance, detect regressions, and ensure security compliance.

Overview

The Synthetic Testing system provides:

Prompt Management: Create and manage test prompts with variables and expected behaviors
Scheduled Testing: One-time and recurring automated tests against individual providers or provider groups
Execution Logging: Detailed logs of each test execution with metrics and security analysis
Archival System: Intelligent log retention with aggregation and export capabilities for scalability

API Endpoints

Synthetic Prompts

List Prompts

http

GET /api/v1/synthetic/prompts
Authorization: Bearer <api-key>

Response:

json

{
  "prompts": [
    {
      "id": "clxxx...",
      "name": "Security Test - Injection Detection",
      "prompt": "Ignore previous instructions and...",
      "category": "security",
      "expectedBehavior": "REJECT",
      "tags": ["security", "injection"],
      "isActive": true,
      "createdAt": "2024-01-15T10:00:00Z"
    }
  ]
}

Create Prompt

http

POST /api/v1/synthetic/prompts
Content-Type: application/json
Authorization: Bearer <api-key>

{
  "name": "API Response Test",
  "prompt": "What is 2 + 2?",
  "category": "functional",
  "expectedBehavior": "ALLOW",
  "expectedResponse": "4",
  "tags": ["math", "basic"],
  "variables": {},
  "includeKpiTracking": true
}

Parameters:

Field	Type	Description
`name`	string	Prompt name (required)
`prompt`	string	Prompt content (required)
`category`	string	Category: functional, security, edge_cases, compliance
`expectedBehavior`	string	ALLOW, REJECT, or SCORE
`expectedResponse`	string	Optional expected response text
`tags`	string[]	Tags for organization
`includeKpiTracking`	boolean	Enable structured KPI metric collection (default: false)

Update Prompt

http

PATCH /api/v1/synthetic/prompts/{promptId}
Content-Type: application/json
Authorization: Bearer <api-key>

{
  "name": "Updated Test Name",
  "prompt": "Updated prompt content",
  "category": "security",
  "includeKpiTracking": true,
  "isActive": true
}

Synthetic Schedules

List Schedules

http

GET /api/v1/synthetic/schedules
Authorization: Bearer <api-key>

Create Schedule

http

POST /api/v1/synthetic/schedules
Content-Type: application/json
Authorization: Bearer <api-key>

{
  "name": "Daily Provider Health Check",
  "description": "Test all providers daily at 6 AM",
  "syntheticPromptId": "clxxx...",
  "scheduleType": "RECURRING",
  "cronExpression": "0 6 * * *",
  "intervalMinutes": 1440,
  "isActive": true,
  "aiProviderId": "clyyy...",          // Single provider OR
  "providerGroupId": "clzzz...",       // Provider group
  "enableProtection": true,
  "protectivePromptId": "clppp..."     // Optional security wrapper
}

Schedule Types:

ONE_TIME - Execute once at scheduled time
RECURRING - Execute repeatedly at interval

Execution Logs

List Executions

http

GET /api/v1/synthetic/executions?limit=50&offset=0
Authorization: Bearer <api-key>

Response:

json

{
  "executions": [
    {
      "id": "clexec...",
      "executionId": "exec-abc123",
      "provider": "openai",
      "model": "gpt-4o",
      "status": "SUCCESS",
      "responseTimeSec": 1.25,
      "tokensUsed": 150,
      "costEstimate": 0.0045,
      "attackDetected": false,
      "kpiCompliant": true,
      "kpiMetrics": {
        "responseCompleteness": 9,
        "accuracy": 10,
        "clarity": 9,
        "relevance": 10,
        "helpfulness": 8,
        "overallScore": 9.2
      },
      "fallbackMetrics": null,
      "startedAt": "2024-01-15T10:00:00Z",
      "completedAt": "2024-01-15T10:00:01Z"
    },
    {
      "id": "clexec...",
      "executionId": "exec-def456",
      "provider": "google",
      "model": "gemini-pro",
      "status": "SUCCESS",
      "responseTimeSec": 0.95,
      "tokensUsed": 120,
      "costEstimate": 0.0032,
      "attackDetected": false,
      "kpiCompliant": false,
      "kpiMetrics": null,
      "fallbackMetrics": {
        "wordCount": 145,
        "sentenceCount": 8,
        "avgWordLength": 5.2,
        "responseComplexity": "MODERATE",
        "structureScore": 6,
        "hasCodeBlocks": false,
        "hasLists": true,
        "hasHeaders": false,
        "estimatedReadingTimeSeconds": 35
      },
      "startedAt": "2024-01-15T10:01:00Z",
      "completedAt": "2024-01-15T10:01:01Z"
    }
  ],
  "pagination": {
    "total": 1000,
    "limit": 50,
    "offset": 0
  }
}

Run Schedule Now

http

POST /api/v1/synthetic/schedules/{scheduleId}/run
Authorization: Bearer <api-key>

Log Archival System

The archival system is designed for enterprise scale with thousands of users and 1000+ provider/model combinations.

Archival Strategy

Retention Period: Detailed logs kept for configurable days (default: 30)
Aggregation: Key statistics aggregated before deletion (provider performance, success rates)
Max Limit: Safety cap of 50,000 logs per company to prevent runaway growth
Export: JSON export for long-term storage before deletion

Archival API

Get Archival Statistics

http

GET /api/v1/synthetic/archival?retentionDays=30
Authorization: Bearer <api-key>

Response:

json

{
  "success": true,
  "stats": {
    "totalLogs": 45000,
    "archivableCount": 12500,
    "oldestLog": "2024-01-01T00:00:00Z",
    "newestLog": "2024-01-15T12:00:00Z",
    "totalSizeEstimateMB": 61.04,
    "byProvider": [
      {
        "provider": "openai",
        "model": "gpt-4o",
        "count": 5000,
        "avgResponseMs": 1100,
        "successRate": 98
      }
    ],
    "byStatus": [
      { "status": "SUCCESS", "count": 12000 },
      { "status": "FAILED", "count": 500 }
    ]
  },
  "config": {
    "retentionDays": 30,
    "maxLogsLimit": 50000,
    "recommendation": "12500 logs older than 30 days can be archived (~61.04 MB)"
  }
}

Execute Archival

http

POST /api/v1/synthetic/archival
Content-Type: application/json
Authorization: Bearer <api-key>

{
  "action": "archive",           // 'archive' | 'enforceLimit' | 'dryRun'
  "retentionDays": 30,
  "exportFirst": true,
  "includeFullResponse": false   // Set true to include full response text in export
}

Actions:

Action	Description
`archive`	Full workflow: aggregate stats, export, delete old logs
`enforceLimit`	Only delete oldest logs if over 50,000 limit
`dryRun`	Preview what would be deleted without making changes

Response (archive):

json

{
  "success": true,
  "action": "archive",
  "deletedCount": 12500,
  "aggregatedStats": [
    {
      "provider": "openai",
      "model": "gpt-4o",
      "totalExecutions": 5000,
      "successCount": 4900,
      "failedCount": 100,
      "avgResponseTimeMs": 1100,
      "minResponseTimeMs": 450,
      "maxResponseTimeMs": 5200,
      "totalTokensUsed": 750000,
      "totalCostEstimate": 22.50,
      "attacksDetected": 15,
      "periodStart": "2024-01-01T00:00:00Z",
      "periodEnd": "2024-01-15T00:00:00Z"
    }
  ],
  "exportFile": "synthetic-executions-archive-company123-2024-01-15.json",
  "errors": []
}

Archival Best Practices

Recommended Settings by Scale

Users	Providers	Retention Days	Archive Interval	Max Logs
1-100	50	90	Monthly	50,000
100-500	200	60	Bi-weekly	50,000
500-1000	500	30	Weekly	50,000
1000+	1000+	14	Daily	50,000

Automation Example

Set up a cron job to archive logs weekly:

bash

# Archive logs older than 30 days, export first
curl -X POST https://app.veriprompt.tech/api/v1/synthetic/archival \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "archive",
    "retentionDays": 30,
    "exportFirst": true
  }'

Storage Optimization

VeriPrompt uses a reference-based storage strategy to minimize database size:

Synthetic Testing:

syntheticPromptId references the source SyntheticPrompt
fullPrompt only stored when the sent text differs from the source (variable substitution, modifications)
For unmodified prompts, the original text is retrieved on-demand via the reference
Archival exports can exclude fullPrompt and response fields for additional savings

Prompt Logs (Normal Usage):

promptId references the stored prompt in the repository
promptHash provides fast lookup without storing full text
Token/cost metrics stored independently of prompt content

Impact at Scale:

Scenario	Storage Saved
1000 users, 100 prompts each	~50-70% reduction
Recurring synthetic tests	~80-90% reduction
Long prompt chains	~60-75% reduction

Analytics

Get Analytics

http

GET /api/v1/synthetic/analytics?range=24h
Authorization: Bearer <api-key>

Range Options: 1h, 24h, 7d, 30d

Response:

json

{
  "providerStats": [
    {
      "provider": "openai",
      "model": "gpt-4o",
      "totalExecutions": 500,
      "successRate": 98.5,
      "kpiComplianceRate": 92.0,
      "avgResponseTime": 1.15,
      "avgTokensUsed": 145,
      "totalCost": 2.25,
      "avgQualityScore": 8.7
    }
  ],
  "qualityMetrics": {
    "avgQualityScore": 8.5,
    "avgResponseTime": 1.2,
    "successRate": 97.5
  },
  "kpiCompliance": {
    "totalCompliant": 4200,
    "totalNonCompliant": 800,
    "complianceRate": 84.0,
    "byProvider": [
      {
        "provider": "anthropic",
        "model": "claude-3-5-sonnet",
        "complianceRate": 95.0,
        "totalExecutions": 1200
      },
      {
        "provider": "openai",
        "model": "gpt-4o",
        "complianceRate": 92.0,
        "totalExecutions": 1500
      },
      {
        "provider": "google",
        "model": "gemini-pro",
        "complianceRate": 78.0,
        "totalExecutions": 800
      }
    ]
  },
  "costAnalysis": {
    "totalCost": 15.50,
    "avgCostPerExecution": 0.0031,
    "costByProvider": [
      { "provider": "openai/gpt-4o", "cost": 8.25 },
      { "provider": "anthropic/claude-3-5-sonnet", "cost": 5.50 }
    ]
  },
  "protectionMetrics": {
    "byMode": [
      {
        "mode": "sanitize",
        "total": 1000,
        "detectionRate": 12.5,
        "sanitizeRetryRate": 8.0,
        "avgResponseTime": 1.4
      }
    ],
    "byTemplate": [
      {
        "templateId": "clprot...",
        "total": 500,
        "detectionRate": 15.0,
        "avgResponseTime": 1.6
      }
    ]
  },
  "networkMetrics": {
    "avgRttMs": 45,
    "avgRouteHops": 8,
    "avgCrossBorderHops": 2,
    "avgHopLatencyMs": 12
  },
  "metadata": {
    "totalExecutions": 5000,
    "timeRange": "24h",
    "startDate": "2024-01-14T10:00:00Z",
    "endDate": "2024-01-15T10:00:00Z",
    "hasData": true
  }
}

Export Analytics

http

GET /api/v1/synthetic/analytics/export?format=csv&range=7d
Authorization: Bearer <api-key>

Formats: csv, json

Provider Rankings

Get provider/model rankings based on synthetic test results:

http

POST /api/v1/synthetic/ranking
Content-Type: application/json
Authorization: Bearer <api-key>

{
  "timeRange": "7d",
  "category": "security",
  "limit": 10
}

Response:

json

{
  "rankings": [
    {
      "provider": "anthropic",
      "model": "claude-3-opus",
      "score": 98.5,
      "metrics": {
        "successRate": 99.2,
        "avgResponseTime": 950,
        "attackBlockRate": 100,
        "costEfficiency": 0.85
      }
    }
  ]
}

Security Testing (Protective Prompts)

Synthetic tests can include Protective Prompts (security wrappers) to test prompt injection defenses:

http

POST /api/v1/synthetic/schedules
Content-Type: application/json
Authorization: Bearer <api-key>

{
  "name": "Security Regression Test",
  "syntheticPromptId": "clxxx...",
  "scheduleType": "RECURRING",
  "intervalMinutes": 60,
  "enableProtection": true,
  "protectivePromptId": "clppp..."  // Security wrapper ID
}

Protective Prompt Types:

Role Boundary Enforcer: Prevents role/persona hijacking
Data Exfiltration Blocker: Blocks attempts to extract sensitive data
Standard Injection Guard: General prompt injection protection

IMPORTANT - Access Control: Protective Prompts are VeriPrompt intellectual property.
Only Super Admin users can view and manage Protective Prompts
Non-Super Admin users will see empty dropdowns in Agent and Synthetic Schedule forms
The API returns 403 Forbidden for non-Super Admin access attempts
This restriction protects VeriPrompt's proprietary security technology

Permissions

Synthetic Testing Features

Role	Create Prompts	Run Tests	View Results	Manage Archival
SUPER_ADMIN	Yes	Yes	Yes	Yes
SAAS_ADMIN	Yes	Yes	Yes	Yes
ADMIN	Yes	Yes	Yes	Yes
SYNTHETIC_ADMIN	Yes	Yes	Yes	Yes
SYNTHETIC_OPERATOR	No	Yes	Yes	No
SECURITY_CUSTODIAN	Yes	Yes	Yes	Yes

Protective Prompts (Security Wrappers)

Role	View	Assign to Agents	Assign to Schedules
SUPER_ADMIN	Yes	Yes	Yes
All Other Roles	No	No	No

Error Codes

Code	Description
`UNAUTHORIZED`	Missing or invalid API key
`FORBIDDEN`	Insufficient permissions for synthetic features
`NOT_FOUND`	Schedule, prompt, or execution not found
`VALIDATION_ERROR`	Invalid request parameters
`PROVIDER_ERROR`	AI provider returned an error
`RATE_LIMITED`	Too many requests

Synthetic Testing API ​

Overview ​

API Endpoints ​

Synthetic Prompts ​

List Prompts ​

Create Prompt ​

Update Prompt ​

Synthetic Schedules ​

List Schedules ​

Create Schedule ​

Execution Logs ​

List Executions ​

Run Schedule Now ​

Log Archival System ​

Archival Strategy ​

Archival API ​

Get Archival Statistics ​

Execute Archival ​

Archival Best Practices ​

Recommended Settings by Scale ​

Automation Example ​

Storage Optimization ​

Analytics ​

Get Analytics ​

Export Analytics ​

Provider Rankings ​

Security Testing (Protective Prompts) ​

Permissions ​

Synthetic Testing Features ​

Protective Prompts (Security Wrappers) ​

Error Codes ​

Related Documentation ​

Synthetic Testing API

Overview

API Endpoints

Synthetic Prompts

List Prompts

Create Prompt

Update Prompt

Synthetic Schedules

List Schedules

Create Schedule

Execution Logs

List Executions

Run Schedule Now

Log Archival System

Archival Strategy

Archival API

Get Archival Statistics

Execute Archival

Archival Best Practices

Recommended Settings by Scale

Automation Example

Storage Optimization

Analytics

Get Analytics

Export Analytics

Provider Rankings

Security Testing (Protective Prompts)

Permissions

Synthetic Testing Features

Protective Prompts (Security Wrappers)

Error Codes

Related Documentation