Review an inference API design for scale

You are reviewing another engineer’s design doc for a machine-learning inference API. Critique and improve it with a focus on distributed systems: clarify product and latency/availability SLOs; estimate throughput and capacity; propose autoscaling, batching, and GPU/accelerator scheduling; handle model loading, versioning, and rollback; design multi-tenant isolation and rate limiting; prevent overload with backpressure, queues, and circuit breakers; define idempotency, retries, and timeouts; mitigate cold starts; specify caching strategy (weights, tokens) and token streaming; plan traffic shaping (canary, A/B), shadowing, and safe rollback; define monitoring, alerting, and error budgets; address privacy, safety filters, audit logs, and cost controls. Provide a high-level architecture and call out key trade-offs.