Katkat
Back

Rank Newly Launched Ads Under Cold Start

PinterestMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

Design a machine learning system to rank and explore cold-start ads launched within the past seven days, balancing marketplace revenue, user engagement, and advertiser budget delivery.

Scenario

On a visual discovery platform like Pinterest, thousands of new ad campaigns are launched daily. Ads active for less than seven days face a severe cold-start problem: they lack sufficient historical click-through (CTR) and conversion (CVR) interaction data. Without accurate predictions, traditional auction rankers (which rank by eCPM = Bid * pCTR * pCVR) either starve new ads of impressions or over-expose low-quality ads, degrading user experience and wasting advertiser budget.

Your task is to design an end-to-end cold-start ranking and exploration system specifically optimized for ads created within the last 7 days.

Requirements

Functional Requirements

  • Feature Extraction & Priors: Leverage visual embeddings, ad copy, advertiser historical performance, campaign metadata, and user/context signals to initialize prediction priors for zero-shot or few-shot ads.
  • Safe Exploration: Formulate an exploration policy (e.g., contextual bandits, variance-based score boosting) that uncovers high-potential ads while enforcing risk guardrails on user negative feedback and advertiser budget burn.
  • Calibrated Scoring: Provide well-calibrated probabilities (pCTR, pCVR) so that exploration bonuses do not distort auction pricing or platform revenue.
  • Smooth Handoff: Define a transition mechanism that shifts an ad from the cold-start model/policy to the mature, high-volume production ranker as impressions accumulate.
  • Delayed Feedback Handling: Account for conversion feedback delays (e.g., 1-day to 7-day attribution windows) in training pipelines.

Non-Functional Requirements & Scale

  • Traffic Scale: 100,000 ad ranking requests per second at peak.
  • Ad Inventory: 50 million candidate ads, with 10%–15% in the cold-start phase (< 7 days old or < 1,000 impressions).
  • Latency: P99 inference latency under 40 ms for ranking.
  • Availability & Freshness: New ads must be eligible for serving within 5 minutes of campaign creation; model priors or bandit states must update in near real-time.

Discussion Points

  1. System Architecture: High-level design covering candidate retrieval, scoring/ranking, exploration mechanism, and streaming event pipelines.
  2. Modeling Approach: Architecture for feature representations (e.g., two-tower embeddings, graph priors across advertiser accounts) and uncertainty estimation.
  3. Marketplace & Calibration: Maintaining auction fairness (GSP/VCG auctions) while injecting exploration exploration terms.
  4. Offline Evaluation & Online A/B Testing: Metrics and offline simulation strategies to validate the cold-start ranker without degrading gross merchandise value (GMV) or CTR.

Hints

Hint 1

Leverage multimodal embeddings (visual and semantic) alongside hierarchical advertiser and category priors to infer initial baseline CTR and CVR.

Hint 2

Formulate exploration using contextual bandits (such as LinUCB or Thompson Sampling) or variance-regularized bidding with an explicit exploration budget cap.

Hint 3

Address delayed conversion labels by using survival analysis, importance weighting, or dual-objective models that separate immediate engagement from long-term conversion.

Follow-up questions

  • How would you prevent low-quality or clickbait ads from taking advantage of high exploration bonuses to drain their budget on low-intent clicks?
  • How does the delayed attribution window (e.g., conversions happening 3–7 days post-click) affect the real-time bandit policy or online gradient updates?
  • When an ad crosses the threshold (e.g., 1,000 impressions), how do you blend the predictions of the cold-start model with the mature production ranker without creating sudden ranking instability?