Katkat
Back

Design Comment Prediction Ranking System

RedditMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenML System Design

Last updated

This question assesses the ability to design an end-to-end ML pipeline for ranking comments by predicted user engagement probability, balancing accuracy, latency, and system scalability.

Problem Statement

Design a machine learning-based ranking system for Reddit comments to display the most engaging comments at the top of a thread. The goal is to maximize user comments per view.

Functional Requirements

  • Rank comments based on the predicted probability of a user posting a reply.
  • Incorporate features such as thread context, user history, and comment content metadata.
  • Serve predictions with sub-100ms latency at high request volumes.

Non-Functional Requirements

  • Scalability: The system must handle millions of daily active users and high-velocity comment streams.
  • Freshness: New comments should be ranked and visible quickly.
  • Fairness: Prevent "echo-chamber" effects and bias against new comments (cold-start).

Deliverables

  1. Problem Formulation: How do you define the label for model training?
  2. Feature Engineering: What features are critical for engagement prediction, and how do you handle them?
  3. Model Architecture: Suggest a model architecture and explain why it fits this use case.
  4. System Design: Describe the inference pipeline, including how you handle real-time vs. offline features.
  5. Evaluation: How do you monitor performance and mitigate bias in production?

Hints

Hint 1

Consider defining the label as a binary action (comment/no comment) or using a proxy like time-spent on the comment thread.

Hint 2

Think about the difference between user-agnostic global popularity signals and personalized features that require a low-latency feature store.

Hint 3

Address the cold-start problem by discussing strategies like epsilon-greedy exploration or feature-based ranking that doesn't rely solely on historical interaction data.

Follow-up questions

  • How would you handle the training-serving skew if the features available during online inference are not identical to those available during offline batch training?
  • If we move to a multi-objective model (e.g., predicting both 'upvote' and 'comment'), how would you combine these scores during the ranking phase?
  • What specific metrics would you monitor in production to detect model degradation, and how would you implement a fallback mechanism if the model service latency spikes?