Katkat
Back

Design an A/B Testing Platform

AffirmSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design a high-scale, low-latency experimentation and A/B testing platform capable of deterministic user bucketing, high-throughput exposure and metric ingestion, and automated statistical reporting.

Scenario

At a modern financial checkout platform, teams frequently experiment with checkout UI flows, underwriting models, promotional APR incentives, and risk thresholds. You are tasked with designing an enterprise-grade A/B testing and experimentation platform that allows product and engineering teams to launch experiments safely, evaluate feature flags at high scale with minimal latency, and compute reliable experiment results.

Requirements

Functional Requirements

  • Experiment Management: Product managers can configure experiments, define target audiences (e.g., geography, device, risk tier), traffic allocations (e.g., 50/50 split), and mutually exclusive experiment layers.
  • Variant Assignment: Given a user_id (or anonymous device ID) and context, deterministically evaluate which variant a user receives. Assignments must be sticky across sessions.
  • Exposure & Metric Logging: Record when an experiment treatment is exposed to a user, and join these exposures with downstream business events (e.g., checkout initiated, loan completed, default rate).
  • Analytics & Statistical Engine: Compute conversion rates, confidence intervals, p-values, and detect Sample Ratio Mismatches (SRM).

Non-Functional Requirements

  • Ultra-low Latency: Flag evaluation must complete in under 10ms (p99) on the server side and must not block critical payment flows.
  • High Availability: 99.999% availability for the assignment evaluation path; if the experiment service degrades, it must fail safely to a default variant.
  • Scale:
    • 50,000 evaluation queries per second (QPS) across all services.
    • 500M business and telemetry events logged per day.
    • Hundreds of concurrent experiments running across web, mobile, and backend microservices.

What Your Design Should Cover

  1. High-Level Architecture: Control plane (experiment admin), Data plane (evaluation SDKs / services), and Analytics plane (metric processing).
  2. Bucketing & Allocation Strategy: Deterministic hashing mechanisms (e.g., salted MurmurHash) for user assignment, handling audience targeting rules, and supporting mutual exclusivity across overlapping tests.
  3. Event Ingestion & Data Pipeline: How exposures and conversion telemetry are captured, buffered, deduplicated, and joined efficiently.
  4. Data Storage & Querying: Schema and technology choices for experiment metadata, event streams, and aggregate reporting (e.g., OLAP systems vs. batch processing).
  5. Trade-offs: Client-side SDK evaluation vs. centralized service evaluation; pre-computation vs. real-time query aggregation; ramp-up stability (ensuring users do not switch variants when traffic expands).

Hints

Hint 1

To achieve sub-10ms evaluation latency and maximum uptime, consider moving the evaluation logic directly into application-embedded SDKs that pull and cache experiment rules asynchronously.

Hint 2

Use a deterministic hash function such as MurmurHash3 with a salt of `user_id + experiment_id` mapped to 10,000 buckets to ensure sticky assignments without requiring centralized state lookups.

Hint 3

Separate exposure logging from variant evaluation: an exposure should only be recorded when the user actually encounters the experimental treatment, not merely when an SDK evaluates a flag.

Follow-up questions

  • How would you detect Sample Ratio Mismatch (SRM) in near real-time, and what are the most common architectural bugs that cause it?
  • How would you implement experiment layers to allow two different teams to run experiments on the same user base without statistical interference?
  • When ramping an experiment from 10% to 50% traffic allocation, how do you ensure that users already assigned to the control or treatment group are never re-assigned to a different group?