Katkat
Back

Design a Merchant Catalog Update Pipeline with Bulk Files and Image Resizing

PinterestSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Tests the ability to architect a scalable, asynchronous catalog ingestion pipeline that balances high-throughput bulk feed processing with low-latency single-item edits while prioritizing price and inventory updates over compute-heavy image processing.

Scenario

Global e-commerce platforms like Pinterest require merchants to sync their product catalogs so that products appear accurately in shopping feeds, search results, and shoppable Pins. Merchants update their catalogs using two primary mechanisms: scheduled bulk catalog feed files (CSV, TSV, or XML containing up to millions of rows) and ad-hoc single-item API updates (e.g., a merchant instantly marking an item out of stock).

Design a scalable catalog update pipeline capable of processing single-item edits and full-catalog bulk feeds for over 1 billion products.

Requirements

Functional Requirements

  • Bulk Feed Ingestion: Ingest and parse bulk feed files containing up to 10 million products per merchant, triggered via scheduled cron or merchant webhook.
  • Real-Time Single Edits: Provide a low-latency API for merchants to update individual product attributes (price, availability, title, etc.).
  • Tiered SLA Processing:
    • Critical business fields (price, availability/stock) must update downstream within 2 minutes.
    • Non-critical fields (descriptions, secondary metadata) and compute-heavy tasks (fetching merchant image URLs, resizing into multi-resolution thumbnails, deduplication) can take up to 2 hours.
  • Conflict Resolution: Safely reconcile conflicts when a merchant issues a single-item API edit while a stale bulk feed is still being processed.

Non-Functional Requirements & Scale

  • Catalog Size: 1 billion total active products across 1 million merchants.
  • Throughput: ~100 million product updates daily, with peak bursts of up to 50,000 updates/second during morning catalog ingestion windows.
  • Idempotency & Cost Efficiency: Feed files often repeat identical data. The pipeline must avoid redundant writes, database updates, and external image downloads.
  • Availability: 99.99% availability for single-item APIs; fault-tolerant worker pipelines for bulk jobs.

What to Cover

  1. End-to-End Architecture: Ingestion service, file chunking, message queues, worker tiers, and image processing pipeline.
  2. Storage & Data Model: Choice of datastore for the primary product catalog vs. raw staging, supporting high write bursts and point lookups.
  3. Prioritization Mechanism: How critical fields (price/stock) bypass the slower media-processing pipeline.
  4. Conflict Resolution: Strategy for handling out-of-order writes and racing bulk vs. single-item updates.
  5. Optimization: Hashing and delta-detection strategies to eliminate redundant downloads and writes.

Hints

Hint 1

Compute a row-level content hash and an image-URL hash during feed parsing to detect deltas and discard unchanged records before touching the database or image pipeline.

Hint 2

Decouple the update pipeline into two separate event streams: a high-priority stream for price/inventory and a low-priority stream for image fetching and resizing.

Hint 3

Use field-level timestamps or merchant-scoped version vectors so a bulk feed generated an hour ago does not overwrite a single-item API update made five minutes ago.

Follow-up questions

  • How would you prevent your image processing fleet from inadvertently DDoS-ing a merchant's origin image server during a large catalog sync?
  • If downstream search and recommendation indexes need to be updated, how do you reflect price and stock changes near-instantaneously without re-indexing entire multi-field product documents?
  • How does your storage and schema design handle semi-structured, merchant-specific custom attributes without requiring continuous database schema migrations?