Katkat
Back

Design a Software Load Balancer

OracleSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This problem tests the ability to architect a scalable, fault-tolerant entry point for distributed systems, focusing on request routing, health monitoring, and concurrency management.

Problem Statement

Design a software-based load balancer (LB) that routes incoming client requests to a pool of backend servers. Your system should handle both L4 (transport layer) TCP connections and L7 (application layer) HTTP/HTTPS requests.

Requirements

  • Functional Requirements:
    • Route traffic based on defined policies (e.g., Round Robin, Least Connections, or Weighted distribution).
    • Periodically perform health checks on backend servers to mark them as healthy or unhealthy.
    • Provide a mechanism for session persistence (sticky sessions) for L7 traffic.
  • Non-Functional Requirements:
    • High availability: The LB should handle its own failure gracefully.
    • Scalability: The system must support thousands of concurrent connections.
    • Latency: The LB should introduce minimal overhead to the request path.

Expected Deliverables

  1. High-level architecture diagram (described in text).
  2. Discussion of how state is managed (e.g., connection tracking, backend server health).
  3. Explanation of the request handling flow for both L4 and L7 traffic.
  4. Strategy for scaling the load balancer itself.

Constraints

Assume the system receives 100,000 requests per second and manages a pool of 50 backend servers.

Hints

Hint 1

Consider using a two-tier approach where an L4 LB handles initial packet routing and a cluster of L7 LBs handles protocol-specific logic.

Hint 2

Think about using a centralized configuration store like Zookeeper or etcd to keep track of backend health status across multiple load balancer instances.

Hint 3

For sticky sessions at the L7 layer, consider using HTTP cookies or hashing source IP addresses to ensure consistent routing.

Follow-up questions

  • How would you handle the 'thundering herd' problem if a large number of backend servers suddenly come back online after a failure?
  • What are the specific trade-offs between using a user-space load balancer (like Nginx) versus a kernel-space solution (like IPVS or eBPF)?
  • How would you implement hot-reloading of backend server lists without dropping active client connections?