Katkat
Back

Design User and Item Embeddings from Long Histories

PinterestMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

Design an end-to-end candidate retrieval system using user and visual-item embeddings capable of processing ultra-long user histories (up to 160,000 interactions) while handling real-time freshness, cold-start items, and privacy constraints.

Scenario

You are designing the primary candidate retrieval system for a large-scale visual discovery platform (similar to Pinterest). Users explore billions of visual items (pins) and generate interaction histories that can span several years and up to 160,000 interactions (clicks, saves, hides, and searches).

Your task is to design an embedding-based retrieval system that generates low-latency representations for both users and multimodal items, matching users to relevant candidates in a shared metric space.

Requirements

Functional

  • Multimodal Item Representation: Encode visual features, text metadata, and graph-based engagement to support both mature items and zero-interaction cold-start items.
  • Long-Sequence User Modeling: Synthesize user preferences from long-term history (up to 160k actions) while staying responsive to real-time session intent.
  • Point-in-Time Correctness: Training datasets must maintain strict causal timelines without future-leakage across long user lifecycles.
  • Privacy & Compliance: Support "Right to be Forgotten" (GDPR/CCPA) by promptly purging user footprints and deleted pins from retrieval stores.

Non-Functional & Scale

  • Scale: 500 million monthly active users, 5 billion candidate items.
  • Latency: Top-K retrieval (e.g., top 1,000 items) must return within P99 < 50ms.
  • Freshness: Item cold-start availability within 15 minutes of upload; user embedding responsiveness to in-session clicks within seconds.
  • Availability: 99.99% availability for candidate generation.

What to Cover

  1. Model Architecture: Two-tower or dual-encoder architecture. How do you compress and attend over 160k interaction events computationally (e.g., hierarchical encoders, memory networks, cluster pooling) without exceeding inference budget?
  2. Training Strategy: Loss function (e.g., sampled softmax, InfoNCE), negative sampling strategies (in-batch, hard negatives), and timeline replay to prevent data leakage.
  3. Serving & Indexing: Architecture for Approximate Nearest Neighbor (ANN) search, handling index version updates without downtime, and serving real-time user query vectors.
  4. Data Lifecycle & Privacy: Deletion propagation pipelines for removed content and user account purges across ANN indexes and downstream caches.

Hints

Hint 1

A standard transformer cannot ingest 160k tokens directly; consider a multi-tiered representation separating static/clustered long-term interests from raw-sequence recent interactions.

Hint 2

For point-in-time correctness during training, construct snapshot states or event replay sequences that only expose interactions prior to the target interaction timestamp.

Hint 3

Decouple cold-start multimodal item embedding generation (purely content-based via vision/text models) from the graph-based refinement applied to mature items.

Follow-up questions

  • How would you prevent popular items from dominating the ANN retrieval results without explicitly hardcoding heuristic penalties?
  • If a user requests account deletion under GDPR, how do you verify and execute unlearning if their past engagement influenced the static embedding representations of items?
  • How do you safely deploy and transition traffic to a new version of the item embedding space without incurring dual-indexing infrastructure costs?