batch.md

Batch Processing

Process thousands or millions of LLM inferences at 50% off serverless pricing. Batch is ideal for evaluations, data processing, and any workload that doesn't require real-time responses.

1. Get an API key

  1. Go to https://www.saas.parasail.io/keys.
  2. Click Create API Key.
  3. Copy the key—it's shown only once.

Store it in your environment:

export PARASAIL_API_KEY="your-api-key-here"

2. Install the batch helper library

pip install openai-batch

The library is 100% compatible with both Parasail and OpenAI. Source code: github.com/parasail-ai/openai-batch.

3. Submit your first batch

import random
from openai_batch import Batch

with Batch() as batch:
    objects = ["cat", "robot", "coffee mug", "spaceship", "banana"]
    for i in range(100):
        batch.add_to_batch(
            model="NousResearch/DeepHermes-3-Mistral-24B-Preview",
            messages=[{"role": "user", "content": f"Tell me a joke about a {random.choice(objects)}"}]
        )
    result, output_path, error_path = batch.submit_wait_download()
    print(f"Batch completed with status {result.status} and stored in {output_path}")

Run it:

PARASAIL_API_KEY=<your-api-key> python3 test_batch.py

4. Monitor via the Batch UI

The Parasail Batch UI lets you upload batches, track status, and download results. Open the Batch tab in the dashboard.

Key features

Next steps