Design a Data Service for Downstream Consumers

Scenario

ByteDance operates numerous core datasets (such as video engagement and creator analytics) that dozens of downstream teams rely on—ranging from real-time recommendation feature stores to batch analytics pipelines. Upstream schema changes frequently break downstream consumers, and direct warehouse access results in resource contention and unpredictable query latency.

Design a multi-tenant Data Access Service that acts as a governed gateway between underlying analytical/operational storage systems and downstream consumer applications.

Requirements

Functional Requirements

• Stable Consumption APIs: Provide flexible read interfaces including single-key projection lookups (e.g., latest creator metrics) and cursor-based batch streaming/pagination for bulk sync.
• Schema Evolution & Versioning: Enforce contract management (e.g., Schema Registry) supporting backward-compatible field additions, deprecations, and projection transforms without breaking downstream callers.
• Data Freshness Evidence: Deliver explicit freshness metadata with every response (e.g., dataset watermark, snapshot timestamp, pipeline lag indicator) to enable downstream reproducibility and validation.
• Governance & Security: Support attribute-based access control (ABAC) and dynamic column-level masking (e.g., PII fields) tailored to consumer credentials.

Non-Functional Requirements

• Low Latency: P99 latency under 30 ms for point/projection lookups.
• High Availability: 99.99% read uptime for the serving layer.
• Scalability: Support 100,000 QPS for point lookups and concurrent multi-gigabyte export streams without impacting primary serving latency.
• Fault Tolerance: Resilient to downstream backpressure and upstream processing delays.

Scale Assumptions

• Core dataset covers 2 billion active entities (e.g., videos, creators).
• 50 TB daily change volume; petabyte-scale cumulative historical data.
• 300+ independent downstream consumer services with differing SLA requirements.

Key Discussion Points

1. Architecture & Storage Tiering: Dual-engine storage layout (e.g., distributed key-value / document cache for low-latency point lookups vs. columnar lakehouse/warehouse for batch exports).
2. API & Cursor Design: Mechanics of cursor tokens (e.g., stateful vs. stateless encoded cursors with partition and offset metadata).
3. Schema Management: How schema validation, projection transformation, and deprecation policies are enforced at the API gateway.
4. Metadata & Watermark Propagation: Capturing and serving source pipeline watermarks and snapshot IDs alongside payload data.