Katkat
Back

How to diagnose traffic and measure relevance?

LinkedInData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to diagnose an unexplained drop in engagement metrics by decomposing a complex user funnel, isolating root causes using multidimensional data, and designing a robust validation framework.

Scenario

You are a Data Scientist at a professional networking platform. On Tuesday morning, your automated monitoring system triggers an alert: the total 'Profile View' count has dropped by 15% day-over-day globally. Your manager asks you to investigate the root cause and propose a strategy to determine if this drop is a data engineering bug or a genuine change in user behavior.

The Data Available

  • Traffic Logs: Page views, clicks, and sessions with dimensions (device type, geography, browser, logged-in status).
  • Engineering Telemetry: Latency, error rates (HTTP 5xx), and client-side heartbeat signals.
  • Experiment Logs: Metadata on all ongoing A/B tests and feature rollouts.
  • User funnel data: Click-through rates (CTR) from the main feed to profile pages.

Your Task

  1. Diagnostic Framework: Describe the systematic steps you would take to isolate the cause. How do you distinguish between an instrumentation error (e.g., tracking pixel failing) and a drop in actual traffic?
  2. Metrics Decomposition: How would you break down the 15% drop to narrow down the scope? Provide specific examples of cohorts or segments you would examine.
  3. Validation: Suppose your analysis points to a specific feature release. How would you design a follow-up experiment or analysis to confirm this feature caused the decline in engagement?

Considerations

  • What role does 'external noise' (e.g., holidays, site outages) play in your diagnosis?
  • How do you handle potential interactions between multiple concurrent experiments?

Hints

Hint 1

Start by checking if the drop is uniform across all traffic sources or localized to a specific dimension, such as a browser version or mobile OS.

Hint 2

Compare the drop in 'Profile Views' against upstream metrics, like 'Feed Impressions' or 'Search Queries', to see if the decline occurred before users reached the profile page.

Hint 3

Use A/B test metadata to check if the treatment group of a recent experiment shows a significantly higher variance in engagement compared to the control group.

Follow-up questions

  • If you conclude the issue is a bug in the logging pipeline, how would you estimate the 'missing' traffic to provide stakeholders with an accurate historical view?
  • How would you adjust your monitoring framework to detect similar drops at a more granular level before they reach a 15% threshold?
  • If the drop is confirmed as genuine, how would you design a 'counter-factual' analysis to estimate the impact on long-term user retention?