batch processing.md

For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs; this page is available as Markdown.

Batch Processing at Scale

Process thousands or millions of inferences offline at 50% off serverless pricing. Batch is ideal for evaluations, data processing, content generation, and any workload that doesn't need real-time responses.

Why batch?

50% off serverless token pricing
Additional 50% off for cached tokens
Up to 50,000 requests and 1000 MB per input file
OpenAI-compatible—use the same batch file format and API
Any HuggingFace model—no dedicated deployment needed
Resumable—submit in one process, monitor in another

Quickstart

import random
from openai_batch import Batch

with Batch() as batch:
    objects = ["cat", "robot", "coffee mug", "spaceship", "banana"]
    for i in range(100):
        batch.add_to_batch(
            model="NousResearch/DeepHermes-3-Mistral-24B-Preview",
            messages=[{"role": "user", "content": f"Tell me a joke about a {random.choice(objects)}"}]
        )
    result, output_path, error_path = batch.submit_wait_download()
    print(f"Batch completed with status {result.status}")

See the batch quickstart for the full guide.

What you can batch

Task	Example models	Guide
Chat completions	Any HuggingFace transformer	Batch full guide
Embeddings	GritLM, GTE-Qwen2	Batch embedding models
Image understanding	Qwen3-VL-8B-Instruct	Batch image understanding
Private models	Any private HuggingFace repo	Batch private models

Pricing

Batch is billed at 50% of serverless pricing, based on parameter count. FP8 models incur no additional cost; FP16 models are 30% more. See full pricing details.