Katkat
Back

How would you test swapping two CTA buttons?

AirwallexData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question evaluates A/B test design, causal inference, funnel metric selection and diagnostic/guardrail thinking within Analytics & Experimentation for a Data Scientist role, and is commonly asked to gauge the ability to reason about user behavior, conversion trade-offs, and risk management in sensitive domains like fintech.

You work on the Airwallex marketing/product landing page. There are two primary CTA buttons:

  • “See a demo”
  • “Get started”

A proposal suggests swapping their positions (e.g., left/right or primary/secondary styling stays the same but the order changes).

Questions:

  1. What user/business problem might this change be trying to solve? Provide 2–3 plausible hypotheses.
  2. Design an experiment (or evaluation plan) to decide whether to ship it:
  • What is the primary success metric and why?
  • What secondary/diagnostic metrics would you track to understand tradeoffs across the funnel?
  • What guardrail metrics matter (especially for a fintech onboarding flow)?
  1. What are key pitfalls and biases to watch for (e.g., novelty effects, cannibalization between CTAs, heterogeneous effects by segment)?

Assume you can run an A/B test on page traffic. Define the unit of randomization and the analysis approach.