operate in production

Operational topics

Rate limits and 429 responses

What it covers: RPM limits per product tier, throttling behavior, and rate-limit increases
Page: Limits and quotas

Quota management

What it covers: GPU quota across Batch and Dedicated, quota errors, and increase requests
Page: Limits and quotas

Autoscaling and replicas

What it covers: Max concurrent requests, target concurrency, smoothing, and replica ranges
Page: Auto-Scaling

Deployment lifecycle

What it covers: Create, pause, edit, resume, and destroy Dedicated Instances
Page: Dedicated Instances

Batch job lifecycle

What it covers: Submission, priority, status polling, cancellation, and terminal states
Page: Batch Quickstart

Batch troubleshooting

What it covers: Validation, file-size limits, stuck jobs, partial failures, and quota errors
Page: Batch Troubleshooting

Retries and idempotency

What it covers: 429 handling, exponential backoff, timeouts, and safe retry limits
Page: Retries and Idempotency

Cost and capacity controls

Use the controls documented elsewhere in the docs to keep workloads predictable:

• Dedicated status page: Each Dedicated Instance has a status page for viewing deployment status and controlling the endpoint. See Dedicated Instances.
• Pause to stop billing: Dedicated Instances bill while active. Pause an endpoint when it isn't needed, and use autoscaling settings to control replica counts. See Auto-Scaling.
• GPU quota: The default GPU quota limits the amount of Batch and Dedicated capacity an organization can consume. See Limits and quotas.
• Batch status: Track batch jobs in the Batch UI or poll status through the API. See Batch Troubleshooting.

Next steps

• Limits and quotas—request rate limits, GPU quota, and quota increase guidance.
• Retries and Idempotency—handle transient errors safely.
• Auto-Scaling—size replicas for production traffic.