Design and Release a Consistent Dark-Store Inventory System

Scenario

On-demand grocery delivery platforms operate dedicated micro-fulfillment centers ("dark stores") stocking thousands of fast-moving grocery SKUs. Because orders are fulfilled within minutes, displaying accurate item availability and preventing overselling is critical. You must design an inventory management service that manages stock levels across dark stores and supports safe, zero-downtime software releases.

Requirements

Functional Requirements

• Availability Check: Query available inventory for SKUs at a designated store.
• Inventory Reservation & Expiry: Place a temporary hold (e.g., 10-minute TTL) on items when a customer enters checkout. Automatically release reservations if the order is abandoned.
• Commit & Release: Transition reservations to committed stock upon successful payment, or release held inventory upon checkout failure or order cancellation.
• Manual Adjustments & Reconciliation: Support real-time stock adjustments (e.g., incoming deliveries, damaged items, picker discrepancies) and periodic batch reconciliation against physical audits.

Non-Functional Requirements

• Strict Consistency: Under no condition should an item be oversold beyond its available on-hand count.
• Idempotency & Resilience: Network timeouts, retries, and concurrent requests must not double-decrement or falsely hold stock.
• Safe Deployments: Support continuous delivery with zero-downtime Kubernetes rollouts and backward-compatible database schema migrations.

Scale Assumptions

• 500 dark stores nationwide.
• ~5,000 active SKUs per store.
• Aggregate read throughput: 25,000 QPS (browsing/catalog integration).
• Aggregate write throughput: 2,500 reservations/updates per second, with high write contention on popular items during peak meal-prep hours.

Discussion Areas

Your design should cover:

1. Data Model & Concurrency: Relational schema design (available, reserved, committed counts) and concurrency control strategies (e.g., conditional updates vs. row-level locking vs. Redis-backed locks).
2. Reservation Lifecycle: State machine for reservations, TTL cleanup, and handling idempotent retries from checkout services.
3. Reconciliation & Auditing: Architecture for resolving drift between physical inventory counts and digital ledger balances.
4. Release & Operations: CI/CD pipeline, Kubernetes deployment pattern (e.g., blue-green or canary with readiness probes), and the expand/contract pattern for schema migrations.