Katkat
Back

Design a distributed rate limiter service

AtlassianSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a scalable, low-latency distributed rate-limiting service that handles high-concurrency requests across multiple microservices.

Problem Statement

Design a distributed rate-limiting service that can be integrated into a large-scale microservices architecture. The service should control the rate of requests per user or API key to prevent abuse and ensure system stability.

Functional Requirements

  • Provide an API for microservices to check if a request is allowed: bool isAllowed(string key, int limit, int windowSizeSeconds).
  • Support fixed-window, sliding-window, or token-bucket algorithms.
  • The limit must be consistent across different application instances behind a load balancer.

Non-Functional Requirements

  • Low Latency: The check must be performed in sub-10ms to avoid impacting end-to-end request time.
  • High Availability: The rate limiter should not be a single point of failure.
  • Scalability: The system must handle 100,000+ requests per second.

Deliverables

  1. High-level architecture: Show how the rate limiter integrates with the service mesh or API gateway.
  2. Data Storage: Choose a storage backend and justify your choice based on performance and consistency requirements.
  3. Handling Concurrency: Explain how to manage race conditions when multiple instances increment the same counter simultaneously.
  4. Scalability Strategy: Discuss how you would handle hot keys (e.g., a single API key making millions of requests).

Hints

Hint 1

Consider the latency overhead of network calls and how local caching might help reduce it.

Hint 2

Evaluate the trade-offs between Redis (as a centralized store) vs. a local in-memory store combined with periodic synchronization.

Hint 3

Think about the difference between 'atomic' operations and distributed locks to ensure counter accuracy under high concurrency.

Follow-up questions

  • How would you handle a scenario where the rate-limiting service goes down? Should you 'fail open' or 'fail closed'?
  • If you use Redis, how do you prevent the rate-limiter data from exhausting the cache memory?
  • How would you adapt the design to support 'burst' traffic patterns versus strict 'steady-state' rate limits?