Katkat
Back

How do you design an A/B experiment?

AnthropicSoftware Engineer · Onsite
HardSoftware EngineerOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design a rigorous A/B experiment for a consumer-facing AI product, focusing on identifying interference, selecting appropriate metrics, and addressing statistical validity.

Scenario

You are a Software Engineer at a company building an AI-powered writing assistant. The product team wants to roll out a new 'Suggested Rewrites' feature that highlights text and proposes improvements. You are tasked with designing an A/B test to determine if this feature increases user engagement (defined as the number of characters edited per session).

Task

Design an experiment framework to evaluate this feature. Your design should address the following areas:

  1. Hypothesis and Metrics: Define the primary success metric, secondary metrics, and guardrail metrics to monitor for negative side effects.
  2. Randomization Strategy: Determine the unit of randomization (e.g., user-level, session-level, or request-level) and explain how you would handle potential interference or network effects.
  3. Sample Size and Power: Describe the factors you would consider when calculating the required sample size and how you would determine the duration of the test.
  4. Bias Mitigation: Identify potential sources of bias (e.g., selection bias, novelty effect, or instrumentation error) and propose strategies to mitigate them.

Example Constraints

  • Assume 1 million daily active users.
  • Assume the feature involves a latency penalty of ~100ms per suggestion triggered.
  • The 'Suggested Rewrites' feature relies on a large language model API call.

Hints

Hint 1

Consider whether the performance impact of the model latency might outweigh the engagement benefits and how guardrail metrics can capture this.

Hint 2

Think about the difference between user-level randomization and session-level randomization when users might interact with the feature across multiple devices.

Hint 3

Discuss how you would account for a 'novelty effect' where users click the feature simply because it is new, rather than because it provides long-term value.

Follow-up questions

  • How would you handle a scenario where the experiment results show a significant increase in engagement but a significant decrease in retention?
  • If you discovered that 'power users' were inadvertently being assigned to the control group due to a caching bug, how would you validate the remaining data?
  • How would you adapt this experiment design if the feature required significant GPU resources, making the cost of running the treatment group high?