Katkat
Back

Design a Revenue Ranking Platform

CreditkarmaMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This problem tests your ability to design a multi-stage ranking system that balances business revenue goals with user relevance, specifically addressing sparse conversion data and long-term user retention.

Scenario

You are tasked with building a revenue ranking system for a financial platform that recommends products (e.g., credit cards, loans) to users. The goal is to maximize the platform's revenue while ensuring the products shown are relevant to the user to maintain trust. Products have different commission structures, and user conversions occur only after a significant delay (days or weeks) and a multi-step funnel (click -> application -> approval -> funding).

Requirements

  • Design the end-to-end ML architecture: how do you collect features, train models, and serve predictions?
  • Address the specific challenges of delayed feedback loops and sparse labels (e.g., many clicks, but very few funding events).
  • Propose a strategy for the ranking objective function: how do you balance "predicted probability of conversion" with "expected revenue"?
  • Explain how to handle the cold-start problem for new financial products or new users.
  • Discuss monitoring strategy: how do you detect drift in both conversion rates and revenue performance?

Assumptions

  • Traffic: 100k daily active users.
  • Data: Millions of historical logs available.
  • Constraints: Ranking must happen in under 100ms.

Hints

Hint 1

Consider a multi-task learning (MTL) approach to model different stages of the funnel simultaneously to mitigate data sparsity.

Hint 2

Address the delayed feedback problem by using censored regression or by incorporating intermediate signals like 'application submission' as proxy labels.

Hint 3

Think about using a multi-armed bandit or an exploration-exploitation strategy to handle the cold-start problem for new products.

Follow-up questions

  • How would you adjust your ranking model if you discovered a significant bias toward high-revenue but low-approval-rate products?
  • How would you ensure that your model remains compliant with fair lending regulations regarding credit product recommendations?
  • If your model needs to be updated with new user behavior data in near real-time, what architectural components would you introduce?