Katkat

Design profit evaluation for loyalty program

Capital OneData Scientist · Technical Screen

Last updated

This question evaluates a data scientist's competency in causal inference, experimental and quasi-experimental design, incremental profit measurement from transaction-level data, statistical power and sample-size estimation, and validation and guardrail metric planning.

A national grocery chain launched a free loyalty card on January 1. You must estimate the 6‑month incremental profit attributable to enrollment. You have: 18 months of household‑level transactions (price, cost, product, store, timestamp), enrollment dates, coupon redemptions, and per‑household acquisition/servicing costs. Design a rigorous evaluation plan: (a) Define the causal estimand (incremental profit per enrolled household) and the exact profit formula with all components (incremental gross margin, discount cost/cannibalization on baseline spend, coupon funding, acquisition cost, servicing cost, fraud/breakage). (b) Propose the primary identification strategy (e.g., randomized holdout vs. observational DiD with matched controls). Write the DiD specification (outcome, treatment, fixed effects) and list the assumptions you will test (parallel trends, composition stability, seasonality, event timing). (c) Specify guardrail metrics (margin rate, substitution, unit economics), and how you’d detect/mitigate selection bias (eligibility rules, IVs/propensity). (d) Power/sample‑size: state your MDE (profit per household) and derive inputs needed (variance of margin dollars, enrollment rate, intraclass correlation). (e) Validation: design an A/A test and a pre‑launch placebo DiD. Be precise about time windows, cohorts, and the decision rule to ship/rollback.