Analytics & Experimentation Interview Questions

Showing 1–30 of 218 results

• AdobeMedium·Data Scientist
  Design Metrics Framework for Adobe Express Performance Evaluation

  This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design Metrics Framework for Adobe Express Performance Evaluation states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  Analytics & ExperimentationOnsite
• AirbnbHard·Data Scientist
  Design and assess an A/B test

  This problem tests your ability to design a statistically sound A/B test for a marketplace platform, including metric selection, power analysis considerations, and handling potential interference effects.

  Analytics & ExperimentationOnsiteA/B TestingExperimental DesignMetrics
• AirwallexMedium·Data Scientist
  Diagnose a dip in approval/conversion rate

  This Analytics & Experimentation prompt evaluates time-series diagnostics, ratio decomposition (numerator/denominator and traffic mix), segmentation and data-quality triage, and causal inference skills at a staff-level applied analytics abstraction.

  Analytics & ExperimentationOnsite
• AirwallexEasy·Data Scientist
  Diagnose conversion-rate time series and CTA swap

  This question evaluates a data scientist's competency in time-series diagnosis, conversion-rate decomposition, causal inference, and A/B experiment design within the Analytics & Experimentation category of Data Science.

  Analytics & ExperimentationOnsite
• AirwallexMedium·Data Scientist
  How would you measure billboard impact?

  This problem tests your ability to design a quasi-experimental framework for measuring the offline impact of an OOH (Out-of-Home) advertising campaign on digital conversion metrics.

  Analytics & ExperimentationOnsiteCausal InferenceMetricsExperimental Design
• AirwallexMedium·Data Scientist
  How would you test swapping two CTA buttons?

  This question evaluates A/B test design, causal inference, funnel metric selection and diagnostic/guardrail thinking within Analytics & Experimentation for a Data Scientist role, and is commonly asked to gauge the ability to reason about user behavior, conversion trade-offs, and risk management in sensitive domains like fintech.

  Analytics & ExperimentationOnsite
• AirwallexMedium·Data Scientist
  Investigate Harassment Surge and Mitigation

  An Airwallex Data Scientist onsite analytics question: a monthly moderation report shows the Harassment violation type spiking, and you must decide whether the surge is real or a measurement artifact, then propose and test mitigations. It probes diagnostic analytics, prevalence-metric design, denominator effects, Simpson's paradox, selection bias, label drift, model calibration, causal inference (ITS/DiD), and experiment design with guardrails.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Analyze an A/B test over last 7 days

  This question evaluates competency in A/B test analysis and experiment design, covering conversion-rate computation, statistical inference (two-proportion testing and confidence intervals), sample-ratio checks, heterogeneity assessment (fixed versus random effects), and power/MDE reasoning.

  Analytics & ExperimentationOnsite
• AmazonMedium·Data Scientist
  Choose Effective Graphs for Data Exploration

  Evaluates chart selection for exploratory data analysis across categorical and numerical variables. Strong answers explain pie charts, distributions, comparisons, relationships, missingness, and when visuals mislead.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Compare Tableau live vs extract and filters

  This question evaluates competency in BI data architecture and large-scale dashboard performance, covering extract vs live connections, refresh and incremental strategies, query pushdown versus in-tool processing, row-level security, filter order-of-operations, performance tuning, and platform migration mapping within the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to assess trade-off reasoning across data freshness, storage, query performance and security for a 100M-row fact table, combining practical application-level decisions (implementation and tuning) with conceptual understanding (filter ordering, calculations, and migration pitfalls) when moving between BI platforms.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Design A/B Test for New Amazon Recommendation Module

  Design A/B Test for New Amazon Recommendation Module evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  Analytics & ExperimentationOnsite
• AmazonMedium·Data Scientist
  Design an A/B Test for Dashboard Engagement Impact

  Amazon data scientist experimentation prompt on designing an A/B test for an energy-usage dashboard redesign, including hypotheses, metrics, guardrails, power, test duration, seasonality, and analysis methods.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Design an operations dashboard with justifications

  This question evaluates a data scientist's ability to define precise operational metrics, justify visualization choices, design interactive drill-downs and alerting, and architect aggregation strategies for high-volume same-day delivery events.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Design and analyze pricing-page A/B test

  This question evaluates data-science competencies in online experiment design and analysis, including statistical power and sample-size estimation, variance-reduction methods, guardrail and SRM monitoring, randomization and contamination handling, sequential monitoring and stopping rules, heterogeneity/subgroup analysis, and result communication within the analytics & experimentation domain. It is commonly asked to assess practical application of statistical inference and experimental rigor in product and business contexts, primarily testing practical application while requiring conceptual understanding of underlying statistical principles and operational trade-offs.

  Analytics & ExperimentationOnsite
• AmazonMedium·Data Scientist
  Diagnose Business Decline Using Key Data Metrics

  Diagnose Business Decline Using Key Data Metrics evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Estimate live sports impact on subscriptions

  This question evaluates a data scientist's competence in causal inference and observational study design within analytics and experimentation, covering skills such as defining treatment and control, selecting outcome metrics and covariates, and choosing identification strategies like matching, IV, and difference-in-differences; it is commonly asked to assess the ability to produce credible causal estimates when randomized experiments are unavailable. It tests both conceptual understanding of identification assumptions (e.g., unconfoundedness, overlap, SUTVA, IV validity) and practical application of matching algorithms, propensity modeling, regression specifications, diagnostics, and staggered-rollout/event-study designs for real-world program evaluation.

  Analytics & ExperimentationOnsite
• AmazonHard·Data Scientist
  Evaluate concession gift-card policy with DID

  This question evaluates a data scientist's competency in causal inference and experimentation design, encompassing difference-in-differences with staggered adoption and event-study diagnostics, data inventory and unit-of-analysis specification, clustering choices, heterogeneity and guardrail analyses, and cost–benefit and optional randomized experiment design. Commonly asked in Analytics & Experimentation interviews because it gauges the ability to justify identification strategies, diagnose violations of parallel trends and propose alternatives, and translate empirical results into business-impact metrics, it sits at the intersection of conceptual understanding and practical application within the Data Science domain.

  Analytics & ExperimentationOnsite
• AmazonMedium·Applied Scientist
  Explain why CTR rises but CVR unchanged

  This question tests the ability to diagnose Simpson's Paradox or composition shifts in funnel metrics when engagement increases without a corresponding lift in final conversion.

  Analytics & ExperimentationOnsiteAnalyticsExperimentationFunnel Analysis
• AmazonMedium·Data Scientist
  Explore Dataset to Assess Quality and Choose Visualizations

  Evaluates fast dataset exploration, data-quality assessment, and visualization planning for business stakeholders. Strong answers profile schema, check missingness and bias, and choose charts aligned to decisions.

  Analytics & ExperimentationOnsite
• AmazonMedium·Software Engineer
  Resolve a Conflict Between Metrics and Customer Feedback

  This question tests your ability to troubleshoot conflicting signals in A/B testing and product analytics, requiring a structured approach to reconciling quantitative data with qualitative sentiment.

  Analytics & ExperimentationOnsiteAnalyticsA/B TestingDecision Making
• AnthropicHard·Software Engineer
  Design a profiling plan for kernels

  This question evaluates competency in performance profiling, experimental design, microarchitectural analysis, statistical methodology, and kernel-level performance engineering.

  Analytics & ExperimentationOnsite
• AnthropicHard·Software Engineer
  How do you design an A/B experiment?

  This problem tests your ability to design a rigorous A/B experiment for a consumer-facing AI product, focusing on identifying interference, selecting appropriate metrics, and addressing statistical validity.

  Analytics & ExperimentationOnsiteA/B TestingExperiment DesignMetrics
• AppleHard·Data Scientist
  Choose Optimal Network Retry Threshold

  This question evaluates competence in statistical analysis, experimental design, and decision-making under trade-offs between playback reliability and battery or radio power consumption.

  Analytics & ExperimentationOnsite
• AppleMedium·Data Scientist
  Design an A/B Test for Homepage Layout Impact

  This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design an A/B Test for Homepage Layout Impact states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  Analytics & ExperimentationOnsite
• AppleHard·Data Scientist
  Evaluate a model and choose metrics

  This question evaluates a data scientist's competency in cost-sensitive model evaluation, handling extreme class imbalance, calibration and threshold derivation, experiment design, and post-launch monitoring and fairness within the Analytics & Experimentation domain.

  Analytics & ExperimentationOnsite
• AtlassianMedium·Data Scientist
  Diagnose Checkout Rate Drop: Steps and Analyses

  Diagnose Checkout Rate Drop: Steps and Analyses evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  Analytics & ExperimentationOnsite
• Capital OneMedium·Data Scientist
  Analyze your favorite app and improve it

  This question evaluates a data scientist's competencies in product analytics, monetization analysis, competitive positioning, metric specification (north-star and guardrails), and experimental design including power and validity considerations.

  Analytics & ExperimentationOnsite
• Capital OneMedium·Data Scientist
  Assess Customer Value with Varied Contract Terms and Costs

  Evaluates subscription contract unit economics across free periods, service costs, marketing costs, churn penalties, and fixed overhead. Strong answers compute customer value, portfolio value, and break-even acquisitions.

  Analytics & ExperimentationOnsite
• Capital OneHard·Data Scientist
  Build a causal ML pipeline end-to-end

  This question evaluates expertise in causal inference and policy learning, covering ATE/CATE estimation, heterogeneous treatment effect methods, off‑policy evaluation, overlap and sensitivity diagnostics, and operational deployment with fairness guardrails in the Analytics & Experimentation domain.

  Analytics & ExperimentationOnsite
• Capital OneHard·Data Scientist
  Define and validate an airline profitability metric

  This question evaluates competency in metric design, business analytics, causal inference, and experimental design by requiring a decomposable airline route profitability metric, operational guardrails, validation via backtests and sensitivity analyses, and a policy experiment plan.

  Analytics & ExperimentationOnsite