Topic

Analytics & Experimentation interview questions

560 Analytics & Experimentation questions from 84 companies, with hints, runnable code and solution walkthroughs.

Questions560

74 easy288 medium198 hard

Your progress0/560

Analytics & Experimentation interview questions by the numbers

Last updated October 28, 2026

Katkat has 560 Analytics & Experimentation interview questions from 84 companies; the most come from Meta (21%), Capital One (11%) and DoorDash (9%). By difficulty: 13% easy, 51% medium and 35% hard. The most common round is Technical Screen (55%). Frequent concepts include Causal Inference, A/B Testing, Experimentation and Metrics.

Companies (84)

Meta21% · 115
Capital One11% · 59
DoorDash9% · 52
Amazon5% · 28
Google4% · 23
Uber4% · 22
TikTok3% · 19
Pinterest3% · 17

Difficulty

Easy13% · 74
Medium51% · 288
Hard35% · 198

Interview rounds

Technical Screen55% · 306
Onsite39% · 218
Online Assessment3% · 16
HR Screen3% · 14
Other1% · 6

Roles

Data Scientist92% · 513
Software Engineer2% · 13
Data Analyst2% · 10
Product Analyst1% · 7
Data Engineer1% · 5

Frequent concepts

Causal Inference13% · 73
A/B Testing12% · 65
Experimentation8% · 42
Metrics7% · 41
Product Analytics5% · 30
Experiment Design3% · 18

Most mentioned in interview experiences

1.Analyze ad watch-time with Excel pivots2 reports
2.Define Ultra success and detect suspicious transactions2 reports
3.Define Ultra success metrics and detect suspicious transactions2 reports
4.Design a flu-shot A/B/n campaign experiment2 reports
5.Design an experiment to evaluate a new ads algorithm2 reports
6.Diagnose a dip in approval/conversion rate2 reports
7.Estimate impact without experiments and pick variant2 reports
8.Evaluate Factors Before Renewing TV-Series Contracts2 reports
9.How would you investigate a metric decline?2 reports
10.Investigate marketplace metrics and experiment rollout2 reports

Counted from 120 candidate interview experiences on Katkat. How we collect and count

Showing 301–330 of 560 results

Design a clustered A/B test with spilloversHard
This question tests the ability to design a cluster-randomized experiment when interactions between users (network interference) violate the stable unit treatment value assumption (SUTVA).
MetaData ScientistAnalytics & ExperimentationHR Screen
Design a clustered notification experiment with guardrailsMedium
This question evaluates experimental-design and causal-inference competencies for cluster-randomized experiments, covering metric selection, power and design-effect calculations, intracluster correlation, spillover detection, sequential monitoring, guardrails, and operational rollout within the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to determine how applicants balance statistical rigor and real-world operational constraints—such as cluster choice, monitoring thresholds, and analysis strategy—testing both conceptual understanding and practical application in production experimentation.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design a feed ads A/B test with guardrailsHard
This question evaluates a data scientist's competency in experimental design, causal inference, statistical power analysis, and product-metric trade-offs within the Analytics & Experimentation domain, and is commonly asked to assess the ability to increase monetization while protecting user and creator experience.
MetaData ScientistAnalytics & ExperimentationOnsite
Design a small-sample launch experiment in EuropeHard
This question evaluates a data scientist's competencies in experimental design, causal inference, estimand specification, power and MDE calculations for heavy-tailed and overdispersed outcomes, interference handling, compliance/IV considerations, and sequential monitoring in field experiments.
MetaData ScientistAnalytics & ExperimentationOnsite
Design an A/B Test for Group Video Calls ImpactHard
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design an A/B Test for Group Video Calls Impact states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design an A/B test for pinned-unread featureHard
This question evaluates proficiency in experiment design and causal inference for product analytics, including choosing the randomization unit, defining eligibility and exposure, selecting primary and guardrail metrics, performing power/MDE and sample-size analysis, and addressing identity cross-contamination and network effects.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design an Experiment to Evaluate New ML ModelHard
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design an Experiment to Evaluate New ML Model states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
MetaData ScientistAnalytics & ExperimentationOnsite
Design an Experiment to Evaluate New Recommendation ModelHard
This problem tests the ability to design an end-to-end A/B test framework for a machine learning model update, focusing on metric selection, causal validity, and post-experiment analysis.
MetaData ScientistAnalytics & ExperimentationOnsite
Design an experiment to evaluate a new ads algorithmHard
This question evaluates a data scientist's competency in experimental design, causal inference, metric selection and trade-offs, randomization and interference reasoning, rollout strategies, and observational methods for algorithm evaluation, including consideration of stakeholder impacts such as users, advertisers, and revenue.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design analytics and experiment for group video callsHard
This question evaluates product analytics and experimentation competencies, including tie-strength and group-affinity measurement, event-level instrumentation design, proxy-product forecasting, participant-cap constraints and operational trade-offs, and experiment design that accounts for network effects and contamination.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design and analyze a group-calls experimentMedium
This question evaluates competency in experimental design, causal inference, metric specification, interference handling, and infrastructure-aware analytics within the Analytics & Experimentation domain for a Data Scientist role.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design and evaluate P2P payments in messagingHard
This question evaluates a data scientist's competency in product analytics, causal inference, experimental design under network interference, monetization measurement (ARPU impact), and risk mitigation for a peer-to-peer payments feature relevant to a Data Scientist role.
MetaData ScientistAnalytics & ExperimentationOnsite
Design and validate ad model launchMedium
This question evaluates competence in experimental design, causal inference, online A/B testing and launch decision frameworks for ad recommendation systems, covering randomization/unit choice, metric definition and guardrails, power and duration estimation, validity checks, monitoring and visualization in the Analytics & Experimentation domain.
MetaData ScientistAnalytics & ExperimentationOnsite
Design and validate an ads feed experimentHard
This question tests the ability to design a rigorous A/B test for a core product feature, covering metric selection, interference handling, and post-experiment analysis under complex constraints.
MetaData ScientistAnalytics & ExperimentationOnsite
Design cluster-randomized test under network effectsHard
This question evaluates a data scientist's experiment design and causal inference skills under interference, covering exposure and estimand definition, graph-based cluster construction, cluster-level randomization and contamination mitigation, metric selection, and design-effect/sample-size calculations.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design experiment for Group Calls with interferenceHard
This question evaluates a data scientist's competency in experimental design and causal inference for networked features, covering metrics selection (primary and guardrail metrics), strategies for mitigating interference via randomization or quasi-experimental approaches, exposure and contamination controls, rollout planning, and pre-registration and power analysis. Commonly asked in Analytics & Experimentation interviews for Data Scientist roles, it assesses reasoning about bias–variance trade-offs, engineering constraints, and validity diagnostics in interconnected user environments, requiring both conceptual understanding of identification and practical application of experiment implementation.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design experiments and diagnose metric changesEasy
This question evaluates product analytics and experimentation skills, including experiment design, metric framework definition, causal reasoning, diagnostic analysis, and marketplace operations trade-offs within the Analytics & Experimentation domain.
MetaProduct AnalystAnalytics & ExperimentationTechnical Screen
Design experiments under network interferenceHard
This question evaluates a data scientist's competence in experimental design and causal inference under network interference, assessing understanding of clustering strategies, exposure and exclusion definitions, contamination control, design effect and sample-size implications, estimation of intra-cluster correlation, and mixed-effects analysis for evolving clusters. Commonly asked in Analytics & Experimentation interviews to probe the ability to preserve internal validity while managing operational feasibility in two-sided marketplaces, it tests both conceptual understanding of interference and practical application skills in design and analysis planning.
MetaData ScientistAnalytics & ExperimentationOnsite
Design marketplace experiments at DoorDashMedium
This question evaluates a candidate's skills in experimental design, product and marketplace analytics, causal inference, and diagnostic data science for multi-sided platforms.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design metrics for violating content exposureHard
This question evaluates competency in metric design and measurement for content safety, including defining exposure metrics, inclusion/exclusion rules, handling label latency and appeals, uncertainty-aware alerting, and A/B test design for a Data Scientist role in the Analytics & Experimentation domain.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design metrics to detect harmful content and fraudEasy
This problem tests the ability to design robust monitoring metrics for detecting fraudulent user activity and policy-violating content, focusing on trade-offs between precision and recall.
MetaData ScientistAnalytics & ExperimentationOnsite
Design pre-launch plan and cluster A/B testMedium
This question evaluates a candidate's skills in experimental design, causal inference, cluster randomization, metric definition and instrumentation, power analysis, and phased rollout decision-making.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Design video-ads experiment and handle null resultsMedium
This question evaluates experimental design, causal inference, statistical power analysis, metric definition, and operational analytics competency within the Analytics & Experimentation domain for a Data Scientist role.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Detect and address Simpson’s paradoxMedium
This question evaluates understanding of aggregation bias and Simpson's paradox, proficiency with stratified or mixed-model estimators, heterogeneity testing, and experiment decisioning within Analytics & Experimentation for a Data Scientist role.
MetaData ScientistAnalytics & ExperimentationOnsite
Detect bots using comment distribution patternsEasy
This problem tests your ability to define metrics for identifying automated behavior and your statistical intuition for validating those metrics against human behavior.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Detect fake accounts and measure their impactEasy
This question tests the ability to design an analytical framework for identifying fraudulent behavior and quantifying the financial or engagement impact of those accounts on a platform.
MetaAnalytics EngineerAnalytics & ExperimentationOnsite
Determine High-Quality Notifications with CTR AnalysisMedium
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine High-Quality Notifications with CTR Analysis states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
MetaData ScientistAnalytics & ExperimentationTechnical Screen
Determine Key Metrics and Design A/B Test for Ad RankingHard
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Key Metrics and Design A/B Test for Ad Ranking states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
MetaData ScientistAnalytics & ExperimentationOnsite
Determine Metrics to Evaluate Notification Impact on UsersMedium
Evaluates product analytics judgment for Facebook push notifications, including how to measure helpful versus harmful notifications, design an event-notification experiment, and balance short-term clicks against retention, fatigue, and opt-out guardrails. Strong answers build a metric hierarchy and use causal measurement.
MetaData ScientistAnalytics & ExperimentationOnsite
Determine Value of Prioritizing Accounts by Unread NotificationsMedium
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Value of Prioritizing Accounts by Unread Notifications states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
MetaData ScientistAnalytics & ExperimentationTechnical Screen