Katkat
Back

Design an Identity Trust Experiment

CoinbaseData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests your ability to design a randomized controlled trial to measure the impact of a new identity verification process on user trust and retention, while addressing potential network effects and platform bias.

Problem Statement

Coinbase is introducing a new 'Verified Identity' badge feature intended to increase trust during peer-to-peer crypto transfers. You need to design an experiment to evaluate if this badge increases the volume of successful transfers between users and improves long-term retention.

The Challenge

  1. Metric Definition: Define a primary metric for 'trust' and a guardrail metric to ensure platform stability.
  2. Experimental Design: The badge creates a potential interference issue: if User A is verified and interacts with User B, the behavior of User B might change, violating the Stable Unit Treatment Value Assumption (SUTVA). Propose an experimental unit (e.g., individual, cluster, or geography) to mitigate this.
  3. Bias Detection: How will you determine if the new verification process disproportionately affects specific user segments (e.g., based on geography or account age)?

Requirements

  • Provide a hypothesis for the experiment.
  • Choose an appropriate randomization strategy.
  • Describe your approach to calculating the Minimum Detectable Effect (MDE) and the duration of the test.
  • Propose a method to handle the interference between users (network effects).

Hints

Hint 1

Consider using a cluster-based randomization strategy, such as geographic regions or social graph communities, to minimize the spillover effect between verified and unverified users.

Hint 2

For trust metrics, think beyond simple conversion rates and consider measuring 'repeat interaction frequency' or 'support ticket reduction' as indicators of increased confidence.

Hint 3

Use a difference-in-differences (DiD) approach if you decide on a cluster-based randomization to account for baseline differences between groups.

Follow-up questions

  • How would you handle the situation where the verification process is computationally expensive or requires manual human review, creating a bottleneck?
  • If the experiment shows a positive impact on volume but a negative impact on new user sign-ups, how would you decide whether to launch the feature?
  • How does your experimental design change if you suspect that 'power users' behave differently than 'casual users' when exposed to the badge?