Katkat
Back

Analyze CTR Data and Train Model

RedditMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteMachine Learning

Last updated

This problem tests the end-to-end workflow of building a CTR prediction model, focusing on feature engineering for high-cardinality categorical data and handling temporal leakage.

Scenario

You are tasked with building a Click-Through Rate (CTR) prediction model for a news feed. You have a dataset consisting of: user_id, article_id, category, timestamp, device_type, and a binary clicked label.

Tasks

  1. Feature Engineering & Leakage: Explain how you would transform the timestamp and article_id into features. Specifically, identify a potential source of data leakage if you were to include the average CTR of an article_id calculated on the entire training set.
  2. Categorical Handling: The user_id and article_id fields have high cardinality (millions of unique IDs). Propose an efficient strategy to handle these categorical features for a Logistic Regression model versus a Gradient Boosted Decision Tree (GBDT) model.
  3. Evaluation: Your model achieves a high AUC-ROC but performs poorly in production (low conversion). What metrics or validation strategies would you look at to diagnose this discrepancy?

Constraints

  • Assume the dataset size is 100 million rows.
  • Ensure the solution accounts for the sequential nature of click data.

Example

Example Input Row: {'user_id': 'u123', 'article_id': 'a456', 'category': 'tech', 'timestamp': '2023-10-01 10:00:00', 'clicked': 0}

  • How would you featurize the timestamp to capture cyclical user behavior?
  • What is the impact of using target encoding on the category feature if the training set is small?

Hints

Hint 1

Consider calculating the mean CTR of an article based only on data points appearing before the current timestamp to prevent look-ahead bias.

Hint 2

For high-cardinality features, consider techniques like frequency-based filtering, embedding layers, or feature hashing.

Hint 3

Temporal validation (e.g., walk-forward validation) is critical when the underlying user behavior evolves over time.

Follow-up questions

  • How would you handle cold-start problems for new articles that have zero impressions?
  • If your model needs to update in near real-time, how would you manage the trade-off between training latency and prediction accuracy?
  • How would you adjust your loss function if the dataset is highly imbalanced (e.g., 0.1% click rate)?