Katkat
Back

Explain concurrency and reliability tradeoffs

NetflixData Engineer · Onsite
HardData EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This problem tests your understanding of balancing thread-safe resource management in a high-concurrency local environment against the systemic requirements of a distributed data pipeline.

Scenario

You are designing a central 'Configuration Registry' for a distributed data ingestion service. This service receives streaming events from thousands of concurrent threads. Each thread must fetch the current configuration parameters before processing an event.

Your task is to design a thread-safe ConfigRegistry class that meets these requirements:

  1. High Concurrency: Minimize lock contention when reading configuration.
  2. Lazy Initialization: The configuration is expensive to fetch from a remote database; it must be fetched only when first requested.
  3. Reliability: If the remote database is unreachable, the registry must provide the last known good configuration (stale read) rather than failing, and attempt an asynchronous refresh.

Requirements

Provide a Python or Java class implementation for ConfigRegistry. Assume a method fetch_remote_config() exists that is slow and prone to timeout.

class ConfigRegistry:
    def get_config(self) -> Config:
        # Implementation required

Discussion Points

  1. Explain the mechanism used for thread-safe lazy initialization.
  2. How does your design handle the tradeoff between providing a 'stale' value versus 'strong consistency' during a database outage?
  3. Describe how you would implement backpressure if the fetch_remote_config() calls begin to saturate the network bandwidth or database connections.

Edge Cases

  • What happens if two threads request the config simultaneously while the registry is empty?
  • How do you prevent 'thundering herd' problems where all threads try to refresh the config at the exact same time when it expires?

Hints

Hint 1

Consider using the double-checked locking pattern or a thread-safe atomic reference to store the current configuration.

Hint 2

Decouple the configuration retrieval from the request flow by using a background worker thread for updates, allowing `get_config` to return immediately.

Hint 3

Think about using a circuit breaker pattern or a simple expiration timestamp to decide when the 'stale' cache needs an asynchronous refresh.

Follow-up questions

  • If this service were scaled across 100 nodes, how would you ensure the 'last known good' configuration is consistent across the entire fleet?
  • How would you modify the system to handle 'rolling configuration updates' where nodes need to adopt new settings at slightly different times to avoid system-wide crashes?
  • If the configuration was sharded by region, how would your locking strategy change to ensure high availability for specific regions even if others are experiencing high latency?