Operational topics

Topic

What it covers

Page

• Rate limits and 429 responses
  RPM limits per product tier, throttling behavior, and rate-limit increases
  Limits and quotas

• Quota management
  GPU quota across Batch and Dedicated, quota errors, and increase requests
  Limits and quotas

• Autoscaling and replicas
  Max concurrent requests, target concurrency, smoothing, and replica ranges
  Auto-Scaling

• Deployment lifecycle
  Create, pause, edit, resume, and destroy Dedicated Instances
  Dedicated Instances

• Batch job lifecycle
  Submission, priority, status polling, cancellation, and terminal states
  Batch Quickstart

• Batch troubleshooting
  Validation, file-size limits, stuck jobs, partial failures, and quota errors
  Batch Troubleshooting

• Retries and idempotency
  429 handling, exponential backoff, timeouts, and safe retry limits
  Retries and Idempotency

Cost and capacity controls

Use the controls documented elsewhere in the docs to keep workloads predictable:

• Dedicated status page: Each Dedicated Instance has a status page for viewing deployment status and controlling the endpoint. See Dedicated Instances.

• Pause to stop billing: Dedicated Instances bill while active. Pause an endpoint when it isn't needed, and use autoscaling settings to control replica counts. See Auto-Scaling.

• GPU quota: The default GPU quota limits the amount of Batch and Dedicated capacity an organization can consume. See Limits and quotas.

• Batch status: Track batch jobs in the Batch UI or poll status through the API. See Batch Troubleshooting.

Next steps

• Limits and quotas—request rate limits, GPU quota, and quota increase guidance.
• Retries and Idempotency—handle transient errors safely.
• Auto-Scaling—size replicas for production traffic.