Katkat
Back

Design config rollout and click aggregation

NetflixData Engineer · Onsite
HardData EngineerOnsiteSystem Design

Last updated

This problem tests your ability to design a global configuration distribution system combined with a high-throughput event processing pipeline to monitor rollout health.

Scenario

Netflix relies on thousands of dynamic configuration toggles (feature flags) to enable canary deployments and A/B tests globally. You are tasked with designing two subsystems:

  1. The Config Distribution Service: A system that pushes configuration updates to millions of client devices. Updates must be versioned, support incremental rollouts (e.g., 5%, 25%, 100%), and allow for instantaneous global rollbacks.
  2. The Telemetry Aggregator: A system that ingests 'impression' and 'click' events from these devices. You need to calculate real-time Click-Through Rates (CTR) per configuration version to identify if a new rollout is degrading user engagement.

Requirements

  • Scale: Handle 10 million concurrent devices, each polling or receiving config updates. Ingest ~500,000 events per second globally.
  • Consistency: Clients must receive consistent configs within a defined window. Rollback must propagate to all clients in under 30 seconds.
  • Storage: Configs are small JSON blobs; telemetry data needs to be aggregated for real-time dashboards and retained for 30 days.

Expected Design Components

  • API Design: Define the interface for clients to fetch configs and send events.
  • Data Model: How do you store config versions and associate events with a specific config version?
  • Architecture: Explain the path from a config change in the admin console to the client, and from the client back to the aggregated CTR metrics.
  • Trade-offs: Discuss how you handle "thundering herd" problems during config updates and how you balance latency vs. consistency in telemetry aggregation.

Hints

Hint 1

Consider using a distributed key-value store or a CDN-backed distribution mechanism to offload the config server.

Hint 2

For telemetry, use a partitioned event stream (like Kafka) and a stream processing engine (like Flink) to calculate windowed aggregates.

Hint 3

Think about how to embed a 'config_version_id' in every telemetry event to ensure every click is attributed correctly to the active configuration.

Follow-up questions

  • How would you modify the system if a configuration change caused a massive spike in client requests (a 'config-induced DDOS')?
  • How would you ensure that telemetry data remains accurate even if some events arrive out-of-order due to network latency?
  • If you needed to perform automated canary analysis, what metrics would trigger an automatic rollback?