Katkat
Back

Keep Merchant Addresses, Social Followers, and Wiki Facts Up to Date

PinterestSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design a high-scale merchant data enrichment platform that ingests, reconciles, and maintains up-to-date business attributes (addresses, social metrics, wiki metadata) across heterogeneous third-party sources with strong provenance and fault tolerance.

Scenario

Modern commerce platforms rely on rich merchant metadata to power merchant profile pages, shopping recommendations, and trust badges. You are tasked with designing an automated enrichment platform that continuously verifies, tracks, and merges merchant details from multiple heterogeneous third-party sources (e.g., social platform follower counts, official business registries, mapping APIs, and Wikipedia/Wikidata knowledge graphs).

Requirements

Functional Requirements

  • Entity Resolution & Linking: Map an internal merchant ID to corresponding verified external source IDs (e.g., matching a domain/brand name to a specific social handle, Wikidata QID, or legal address entry).
  • Periodic & Event-Driven Ingestion: Collect metric snapshots (e.g., social follower counts updated daily), semi-static data (e.g., headquarters address, founding year, parent company updated monthly), and react to merchant-initiated re-sync requests.
  • Provenance & Lineage: Every enriched attribute field must retain its source attribution, extraction confidence score, and timestamp.
  • Resilience & Partial Failures: If one external upstream API is throttled or down, the system must continue serving the latest known good attributes without failing the entire profile enrichment.
  • Downstream Serving: Provide low-latency read APIs for internal services (e.g., Merchant Profiles, Shopping Pin ranking) to fetch current merchant states and historical snapshots.

Non-Functional Requirements

  • Scale: 50 million merchant profiles. Around 5 million "active" merchants require daily social follower refresh; the remainder refresh weekly.
  • Freshness SLA: High-priority social metrics updated within 24 hours of capture; address/fact updates reflected within 7 days.
  • Read Latency: P99 latency < 25 ms for internal profile lookups at 20,000 reads/second.
  • Auditability: Complete historical changelog preserved for compliance and debugging.

What Your Design Should Cover

  1. Data Model & Schema: Representing dynamic attributes, source provenance, confidence scores, and time-series snapshots.
  2. System Architecture: Ingestion workers, rate-limiting queues, scheduling, entity matching service, and data reconciliation logic.
  3. Conflict Resolution & Merging: Strategy for handling conflicting data (e.g., mismatched addresses across two official registries).
  4. Failure Modes: Dealing with upstream API rate limits, breaking schema changes, and poison pills.

Hints

Hint 1

Separate the raw ingestion and entity-linking pipeline from the served materialized view so downstream reads are decoupled from upstream API latency and failures.

Hint 2

Model merchant attributes using an entity-attribute-value (EAV) or document approach with explicit metadata envelopes containing source ID, timestamp, and confidence score.

Hint 3

Implement a priority-aware, token-bucket rate-limiting job queue per external provider to maximize API quota utilization while preventing IP bans.

Follow-up questions

  • How would you detect and alert on subtle upstream hallucinations or schema changes (e.g., an external API starts returning follower counts formatted as strings with 'M' and 'K' suffixes instead of integers)?
  • How does your design handle conflicting addresses between a user-submitted claim and an authoritative government registry?
  • If an upstream source enforces a hard limit of 50 requests/second across our entire infrastructure, how do you dynamically prioritize which 50 merchants get refreshed every second?