retries and idempotency.md

Retries and Idempotency

Production clients should expect occasional transient errors and retry them safely. This page covers when Parasail returns 429 Too Many Requests, how to back off, how to set timeouts, and what you can and can't assume about retry idempotency.

When you'll see 429 Too Many Requests

A 429 means a request was throttled and should be retried later. There are two common causes on Parasail:

In both cases the right response is the same: wait, then retry with backoff.

Exponential backoff

Retry transient errors (429, and 5xx server errors) with exponential backoff and jitter, rather than retrying immediately in a tight loop. Cap the number of attempts so a request can't retry forever.

The OpenAI Python SDK already retries certain errors automatically; you can configure max_retries on the client and still wrap calls for full control:

import random
import time

from openai import OpenAI, APIStatusError, APITimeoutError, APIConnectionError

client = OpenAI(
    base_url="https://api.parasail.io/v1",
    api_key="<PARASAIL_API_KEY>",
    timeout=30,        # seconds; fail fast instead of hanging
    max_retries=0,     # disable built-in retries so we control backoff here
)

def chat_with_backoff(messages, model, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(model=model, messages=messages)
        except (APITimeoutError, APIConnectionError):
            retryable = True
        except APIStatusError as e:
            # Retry on throttling and server errors; fail fast on 4xx like 400/401/404.
            retryable = e.status_code == 429 or e.status_code >= 500
            if not retryable:
                raise
        if attempt == max_attempts - 1:
            raise
        # Exponential backoff with full jitter: 1s, 2s, 4s, ... plus randomness.
        sleep = min(2 ** attempt, 30) * (0.5 + random.random() / 2)
        time.sleep(sleep)

response = chat_with_backoff(
    messages=[{"role": "user", "content": "Hello"}],
    model="<your-model>",
)
print(response.choices[0].message.content)

Setting timeouts

Always set a client-side timeout so a stalled connection fails fast and can be retried, instead of hanging indefinitely. The example above sets timeout=30 on the client; you can also override it per request:

client.with_options(timeout=60).chat.completions.create(...)

Choose a timeout that comfortably exceeds your model's expected latency. Long generations and large batch-style requests need more headroom than short completions.

Safe retries and idempotency

Retrying is only safe if a duplicated request can't cause unintended side effects. For Parasail's inference endpoints (chat completions, embeddings), a retried request simply produces another completion—there is no state to corrupt—so retrying a failed or timed-out call is generally safe. The main caveat is that you may be billed for both a request that actually succeeded server-side and its retry, if the original response was lost in transit.

Parasail exposes an OpenAI-compatible API, and the repository does not document an idempotency-key mechanism. Treat retries as not guaranteed idempotent: a retry can result in a second execution rather than returning the original response. To limit duplicate work:

The public Parasail docs do not define an idempotency-key header. If your workflow cannot tolerate duplicate execution, deduplicate on the client side with your own request IDs and only retry operations that are safe to run again.

Next steps