Katkat
Back

Diagnose metric anomalies and evaluate new algorithm

AirtableData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to systematically investigate a sudden drop in a key product metric and design a robust A/B test to evaluate the impact of a new recommendation algorithm.

Scenario

You are a Data Scientist at Airtable. The product team reports that the 'Views Created' metric (the number of new grid, calendar, or gallery views added by users) has dropped by 15% week-over-week. Meanwhile, the Engineering team has just rolled out a new 'Smart View Suggestions' algorithm to a subset of users.

Task 1: Metric Investigation

How would you structure a diagnostic investigation to determine if the drop is caused by the new algorithm or an external factor? List the specific data cuts or segments you would examine first.

Task 2: Experiment Design

You need to validate whether the 'Smart View Suggestions' algorithm is the root cause. Propose an A/B testing framework to evaluate the impact of this algorithm on the 'Views Created' metric.

Your response should include:

  1. The definition of your treatment and control groups.
  2. The primary metric and any secondary/guardrail metrics you would track.
  3. How you would handle potential cannibalization or network effects if users in the treatment group interact with users in the control group within shared workspaces.

Constraints

  • Assume the product has millions of active users.
  • Assume you have access to event-level clickstream data and workspace metadata.

Hints

Hint 1

Start by investigating whether the drop is isolated to specific cohorts, such as power users vs. new users, or specific browser/platform types.

Hint 2

For the experiment design, consider whether randomizing at the user level or the workspace level is more appropriate given that multiple users can collaborate in the same Airtable base.

Hint 3

Think about 'Guardrail metrics'—what metrics should stay neutral or improve to ensure the feature isn't just increasing view creation at the cost of overall workspace performance or retention?

Follow-up questions

  • If the experiment shows a positive lift in view creation but a slight decline in retention, how would you decide whether to roll out the feature?
  • How would you account for 'novelty effects,' where users interact with the new feature simply because it is new, not because it is useful?
  • How would you measure the long-term impact on user workflow efficiency beyond just the count of views created?