Katkat
Back

Design a Low-Latency GPU Inference Service

AnthropicSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

This problem tests the ability to design a high-throughput, low-latency inference architecture for large language models, focusing on hardware utilization and bottleneck mitigation.

Design a Low-Latency GPU Inference Service

You are tasked with building a scalable inference service for a Large Language Model (LLM) that receives requests from a user-facing application. The service must support sub-second latency for short prompts while maintaining high throughput under peak loads.

Requirements

  • Functional: Provide a REST or gRPC endpoint to accept text prompts and return model completions.
  • Non-functional: Support 1,000 requests per second (RPS) with a P99 latency of under 500ms.
  • System Constraints: The model is too large to fit on a single GPU; you must account for model parallelism (e.g., tensor or pipeline parallelism).

Your design should address:

  1. Serving Strategy: How will you manage GPU memory and VRAM to handle concurrent requests? Consider batching strategies (e.g., Continuous Batching).
  2. Infrastructure: How do you handle autoscaling of GPU nodes versus CPU-based load balancers?
  3. Latency Optimization: How do you minimize overhead between the network request and the GPU execution context?
  4. Failure Modes: How does the system handle a GPU node crash during an ongoing inference stream?
  5. Monitoring: What key metrics are critical for identifying performance degradation at the hardware level?

Hints

Hint 1

Consider the difference between static batching and continuous batching in the context of KV cache memory management.

Hint 2

Discuss how load balancing should be aware of the 'stateful' nature of long-running inference jobs.

Hint 3

Think about how to isolate the request-handling logic (CPU-bound) from the tensor computation logic (GPU-bound).

Follow-up questions

  • How would you modify your design if you needed to support multi-tenant workloads with different priority levels?
  • What architectural changes are required to support speculative decoding to improve latency for longer generation tasks?
  • If you observe 'tail latency' spikes during peak traffic, how would you diagnose whether the bottleneck is network I/O, GPU compute, or memory bandwidth?