Katkat
Back

Build and evaluate an order prediction model

AirbnbData Scientist · Technical Screen
MediumData ScientistTechnical ScreenMachine Learning

Last updated

This question tests the ability to design a robust end-to-end machine learning pipeline for binary classification, focusing on preventing temporal data leakage, managing class imbalance, and aligning model thresholds with business objectives.

Scenario

You are tasked with building a machine learning model to predict whether a guest will book a specific listing within 7 days of viewing it on our platform. The dataset contains user interaction logs, listing attributes, and historical booking labels.

The Task

  1. Validation Strategy: Propose a cross-validation strategy that respects the temporal nature of the data to avoid leakage.
  2. Feature Engineering: Describe three distinct categories of features you would derive from the clickstream data and explain how you would handle potential look-ahead bias.
  3. Business Optimization: The business team requires high precision (minimizing false positives) to avoid sending irrelevant promotional discounts. If the dataset has a 2% conversion rate, how would you approach threshold selection and metric optimization?
  4. Drift Monitoring: Describe two metrics you would track in production to detect model performance degradation due to changes in user behavior or seasonality.

Example Context

  • Input: UserID, ListingID, InteractionTimestamp, IsBooked(Target)
  • Constraints: Avoid using information from the future (e.g., "Total bookings by user" must be restricted to the time before the current interaction).

Hints

Hint 1

Instead of random K-fold, consider a 'sliding window' or 'expanding window' validation approach where training sets are strictly chronologically before the test set.

Hint 2

For class imbalance, consider techniques like cost-sensitive learning or stratified sampling, and evaluate using Precision-Recall curves rather than accuracy.

Hint 3

Think about how the 'distribution' of features (e.g., click-through rates) might shift if the platform changes its UI or if a new marketing campaign is launched.

Follow-up questions

  • How would you handle cold-start problems for new listings that have no historical booking data?
  • If the business asks to maximize revenue rather than simple booking counts, how would you modify the model's objective function?
  • What specific steps would you take to retrain the model if you detect feature drift in the 'user search intensity' feature?