Batch Processing

Process thousands of prompts at 50% cost savings by leveraging provider-native batch APIs. Submit once, VeriPrompt handles provider selection, splitting, and result aggregation.

Why use Batch Processing?

Benefit	How it works
50% cost savings	OpenAI, Anthropic, and Gemini all offer 50% discounts for batch workloads
Unified interface	One JSONL format works with any provider — no need to learn each batch API
Smart provider selection	Uses your configured providers, routing policies, and provider groups
Multi-provider splitting	Split large batches across providers for speed or cost optimization
Centralized billing	Batch costs tracked alongside real-time usage in your billing dashboard

Quick Start

1. Navigate to Batch Processing

Open the sidebar and click Batch Processing under the Ranger section. You'll see the batch dashboard.

2. Submit a Batch

Click New Batch to open the submit form. You have two options:

Option A: File Upload

• Prepare a JSONL file with one prompt per line
• Drag and drop the file onto the upload area

Option B: Inline Editor

• Switch to the "Inline Editor" tab
• Type or paste your JSONL items directly

Each line must be a JSON object with id and prompt:

{"id": "q1", "prompt": "Summarize this article: ..."}
{"id": "q2", "prompt": "Translate to French: Hello, how are you?"}
{"id": "q3", "prompt": "Extract entities: {{text}}", "variables": {"text": "Apple was founded by Steve Jobs."}}

3. Configure (Optional)

Click Advanced Configuration to set:

• Provider — Force a specific provider (e.g., OpenAI, Anthropic)
• Quality Tier — Choose between cheap, fast, quality, or secure
• Model — Specify an exact model (e.g., gpt-4o-mini)
• Split Strategy — How to distribute across providers
• Webhook URL — Get notified when the batch completes
• PII Sanitization — Strip personal data before sending to providers

4. Monitor Progress

After submission, you're taken to the detail view where you can:

• Watch the progress bar update in real-time
• See item-by-item completion stats
• View cost estimates and actual costs
• Cancel the batch if needed

5. Download Results

Once complete, results appear in the table. Click Download JSONL to get all results as a file.

Variable Substitution

Use placeholders in your prompts and provide values in the variables field:

{"id": "email-1", "prompt": "Write a professional email to {{name}} about {{topic}}", "variables": {"name": "John Smith", "topic": "Q3 results"}}
{"id": "email-2", "prompt": "Write a professional email to {{name}} about {{topic}}", "variables": {"name": "Jane Doe", "topic": "product launch"}}

Split Strategies

When your company has multiple AI providers configured, you can split batches across them:

Single (default)

All items go to the best available provider. Simplest option, good for most use cases.

Round-Robin

Items are distributed evenly. Use this when you want to balance load or compare providers.

Cost-Optimal

More items are sent to cheaper providers, fewer to expensive ones. Best for maximizing savings on large batches.

Provider Selection

Providers are selected in this priority order:

1. Explicit provider hint — If you specify a provider in Advanced Configuration
2. Provider Group — If your routing policy references a provider group
3. Routing Policy — RankingEngine scores and filters based on your policy's constraints
4. Company providers — Your company's configured providers (ProviderModelConfig)
5. Platform providers — VeriPrompt's default providers as fallback

Only providers with batch API support are considered (OpenAI, Anthropic, Gemini/Google).

Limits and Quotas

Limit	Default	Configurable
Max items per batch	50,000	BATCH_MAX_ITEMS
Max file size	100 MB	BATCH_MAX_FILE_SIZE_MB
Max concurrent active jobs	10	BATCH_MAX_CONCURRENT_JOBS
Result retention	30 days	BATCH_RESULTS_TTL_DAYS

API Access

All batch operations are available via REST API. See the Batch Processing API Reference for endpoints, request/response schemas, and code examples.

Use Cases

Evaluation Runs

Test hundreds of prompts against different models to compare quality and cost.

Data Processing Pipelines

Process CSV exports through AI for classification, extraction, or summarization.

Content Generation

Generate product descriptions, email templates, or translations in bulk.

Research & Analysis

Analyze large document sets with consistent prompt templates using variable substitution.