Katkat
Back

Design a Distributed Rate Limiter

ByteDanceSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a scalable, low-latency rate limiting service that functions across multiple distributed application servers.

Distributed Rate Limiter

Design a rate-limiting service that can be used by various microservices within a large-scale architecture to prevent abuse and manage traffic spikes. The system must enforce rate limits (e.g., "100 requests per minute") per user ID across a cluster of servers.

Requirements

Functional Requirements:

  • Clients can request authorization to proceed with an action based on a unique identifier (e.g., user_id or api_key).
  • The system must accurately count requests and deny those exceeding the configured threshold.

Non-Functional Requirements:

  • Low Latency: The rate limiter must add minimal overhead to the request flow (target < 10ms).
  • High Availability: If the rate limiter service fails, it should fail open (allow traffic) or maintain consistency, depending on the strategy.
  • Scalability: The system must handle millions of requests per second across geographically distributed clusters.

Deliverables

  • Define the high-level API for the rate limiter client.
  • Explain the data storage and choice of algorithm (e.g., Token Bucket, Fixed Window, Sliding Window).
  • Describe the architecture to ensure low latency and consistency in a distributed environment.
  • Discuss handling clock skew and race conditions during high concurrency.

Hints

Hint 1

Consider the performance impact of network round-trips if the rate limiter is a standalone service versus an embedded library.

Hint 2

Evaluate the trade-offs between using a centralized store like Redis versus a local in-memory store on each node.

Hint 3

Think about how to handle the 'thundering herd' problem or race conditions when updating counts in a distributed cache.

Follow-up questions

  • How would you handle a scenario where a user makes rapid requests across different geographical regions simultaneously?
  • If you are using Redis for storage, how would you design for high availability and prevent it from becoming a single point of failure?
  • How would you dynamically update rate limit thresholds for specific users without restarting the application services?