Design scalable, highly available GenAI serving
OracleSoftware Engineer · Onsite
Last updated
This question evaluates understanding of scalable, highly available generative AI inference platforms and associated competencies in distributed systems, ML model serving, autoscaling and GPU scheduling, global request routing, model/version management, stateful dependency handling, observability, and rate limiting.
Design the deployment of a generative AI model for high scalability and high availability. Describe the inference serving architecture, request routing, autoscaling (including GPU scheduling), multi-region failover, model versioning and rollout, stateful dependency management (tokenizer, embeddings, caches), observability, rate limiting, and strategies to meet latency/throughput SLOs under traffic spikes and failures.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.