Katkat
Back

Implement resilient LLM provider pool

PerplexitySoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenML System Design

Last updated

This English-language summary evaluates a candidate's ability to design a resilient, multi-provider LLM client library encompassing provider management, governance (rate limiting and quotas), resilience mechanisms (health checks, timeouts, circuit breakers), routing (cost- and latency-aware load balancing, retries and fallbacks), observability, security, and concurrency. Commonly asked in the ML System Design domain to assess architectural decision-making and trade-off reasoning for availability, cost, latency, and operational robustness, the question targets practical application and architectural-level thinking rather than purely conceptual or low-level implementation details.

Implement a resilient client library that routes requests across multiple external LLM providers. Support provider registration, capability mapping (models and context length), per-provider rate limiting and quotas, health checks, timeouts, and circuit breakers; cost- and latency-aware load balancing; retry and fallback strategies when a provider is degraded or down; observability (metrics, logs, tracing); secure key management; and thread-safe concurrency. Describe interfaces, data structures, and failure handling.