Katkat
Back

Design Identity & Trust Experiment

CoinbaseData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design a robust A/B test for a high-stakes identity verification flow, focusing on balancing security metrics with user acquisition and handling potential network interference.

The Problem

Coinbase is planning to introduce a new biometric liveness check (e.g., a selfie scan) to its identity verification flow. The product team believes this will reduce fraudulent sign-ups by 40% but fears it might increase user drop-off during onboarding. You are tasked with designing an A/B experiment to validate this feature.

Requirements

  1. Metric Selection: Define a primary metric for the experiment and at least two guardrail metrics to ensure the feature doesn't hurt business health.
  2. Experiment Design: Propose the randomization unit (e.g., user vs. IP address vs. device ID) and explain your reasoning, specifically regarding potential interference between groups.
  3. Power Analysis: If the baseline sign-up rate is 20% and the business requires detecting a minimum drop of 0.5% (absolute) in the conversion rate, what factors must be considered to determine the required sample size?
  4. Bias Mitigation: How would you account for the fact that biometric checks may have varying success rates across different demographics or device types?

Considerations

  • Fraudulent actors often use multiple accounts; consider how this behavior might bias your experiment results if you randomize by user.
  • The verification flow is a bottleneck; if the new feature fails, you risk losing the entire cohort of potential new users.

Hints

Hint 1

Consider that the primary goal is a trade-off between security (fraud reduction) and growth (conversion), so your metrics should reflect both a top-line business outcome and a risk-based metric.

Hint 2

If you randomize by user, but a fraudulent user creates multiple accounts to test the system, your results may suffer from contamination; think about clustering or randomization units that mitigate this.

Hint 3

For power analysis, remember that the variance of a proportion depends on the mean, so estimate your sample size using the variance of the baseline conversion rate while accounting for the expected effect size.

Follow-up questions

  • How would you measure the 'False Rejection Rate' of this new biometric check without a ground-truth label for all users who dropped off?
  • If the experiment shows a positive impact on fraud but a 2% drop in conversion, what framework would you use to decide whether to ship the feature?
  • How would you handle a situation where the treatment effect varies significantly between high-risk regions and low-risk regions?