Design profit evaluation for loyalty program
Capital OneData Scientist · Technical ScreenLast updated
This question evaluates a data scientist's competency in causal inference, experimental and quasi-experimental design, incremental profit measurement from transaction-level data, statistical power and sample-size estimation, and validation and guardrail metric planning.
A national grocery chain launched a free loyalty card on January 1. You must estimate the 6‑month incremental profit attributable to enrollment. You have: 18 months of household‑level transactions (price, cost, product, store, timestamp), enrollment dates, coupon redemptions, and per‑household acquisition/servicing costs. Design a rigorous evaluation plan: (a) Define the causal estimand (incremental profit per enrolled household) and the exact profit formula with all components (incremental gross margin, discount cost/cannibalization on baseline spend, coupon funding, acquisition cost, servicing cost, fraud/breakage). (b) Propose the primary identification strategy (e.g., randomized holdout vs. observational DiD with matched controls). Write the DiD specification (outcome, treatment, fixed effects) and list the assumptions you will test (parallel trends, composition stability, seasonality, event timing). (c) Specify guardrail metrics (margin rate, substitution, unit economics), and how you’d detect/mitigate selection bias (eligibility rules, IVs/propensity). (d) Power/sample‑size: state your MDE (profit per household) and derive inputs needed (variance of margin dollars, enrollment rate, intraclass correlation). (e) Validation: design an A/A test and a pre‑launch placebo DiD. Be precise about time windows, cohorts, and the decision rule to ship/rollback.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.