Katkat
Back

Design a Read- and Write-Heavy Product Catalog Service

Walmart LabsSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a scalable, highly available product catalog platform supporting both high-frequency merchant updates and massive shopper read traffic while maintaining data integrity and search consistency.

Scenario

Design a multi-merchant product catalog service for an enterprise e-commerce platform like Walmart. The platform hosts hundreds of millions of items and must handle concurrent bulk updates from automated merchant feeds (price adjustments, inventory updates, metadata edits) alongside millions of shoppers browsing, searching, and filtering items in real time.

Requirements

Functional Requirements

  • Item Lifecycle Management: Merchants can create, update (partial and full updates), and withdraw/delist products.
  • Catalog Retrieval: Shoppers can fetch single product detail pages (PDP) and browse paginated listings filtered by category, merchant, price, and status.
  • Search & Filter Synchronization: Merchant updates (such as price or delisting) must be reflected across search indices and browse pages within an acceptable time window.
  • Concurrency & Versioning: Concurrent writes to the same item must be resolved deterministically without lost updates.

Non-Functional Requirements

  • High Availability & Scalability: 99.99% availability for customer-facing reads; high write availability for merchant integrations.
  • Low Latency: Read latency < 30ms at p99 for single-item lookups; < 100ms for paginated listing queries.
  • Eventual Consistency with Bounds: Product changes must propagate to read models and caches within 1–2 seconds. Delisted items must be hidden promptly.

Scale Assumptions

  • Catalog Size: 200 million distinct SKUs.
  • Read Traffic: 100,000 QPS peak (browsing, detail pages, search).
  • Write Traffic: 15,000 updates/second peak (automated price scrapers, bulk inventory syncs, seller portal).

Key Areas to Address

  1. Data Modeling & Storage Selection: Define the authoritative write store vs. the read-optimized store (e.g., CQRS with document/relational store and Elasticsearch/OpenSearch).
  2. API Design: Define endpoints for partial item updates, product reads, and paginated searches.
  3. Concurrency & Ordering: Address race conditions (e.g., out-of-order webhook delivery, simultaneous price and stock edits) using versioning or optimistic locking.
  4. Caching & Invalidation Strategy: Multi-tier caching (CDN, distributed cache) and invalidation or TTL strategies, especially for rapid item withdrawals.
  5. Resilience & Failure Recovery: Handling CDC (Change Data Capture) lag, broker failure, and recovery when re-indexing large segments of the catalog.

Hints

Hint 1

Consider applying Command Query Responsibility Segregation (CQRS) to separate the authoritative normalized write database from read-optimized search indices and document stores.

Hint 2

Use optimistic concurrency control (e.g., item version numbers or monotonic sequence IDs) in merchant write requests to avoid lost updates from concurrent updates.

Hint 3

Decouple search indexing and cache invalidation from the synchronous write path using an asynchronous event-driven pipeline (like Kafka with Change Data Capture).

Follow-up questions

  • How would you handle a merchant submitting a bulk feed file containing 500,000 SKU updates without starving real-time manual updates from the seller dashboard?
  • If your search index falls 30 seconds behind the authoritative store during a high-traffic sales event, how do you prevent users from adding out-of-stock or delisted items to their carts?
  • How would you structure cursor-based pagination for listing views where items frequently change rank or get delisted during user browsing?