Katkat
Back

Design a High-Volume Healthcare Data Ingestion Pipeline

OracleBackend Engineer · Onsite
MediumBackend EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a resilient, high-throughput ETL pipeline for sensitive and messy healthcare data, focusing on fault isolation, data integrity, and architectural scalability.

Scenario

You are tasked with building an ingestion pipeline for a healthcare platform that receives large batches of patient records from hundreds of disparate hospital partners. The system must handle 500 million records per day with the following specific challenges:

  1. Data Quality: Incoming payloads are often malformed, contain duplicates, or are out-of-order versions of previous records.
  2. Compliance: Data is PII/PHI sensitive and must be encrypted at rest and in transit. Access logs must be immutable.
  3. Reliability: A failure in parsing one provider's data should not block the entire pipeline.
  4. Traceability: You must support "replayability," allowing the system to re-process historical raw data if parsing logic is updated.

Requirements

  • Design the architecture from ingestion to a final 'curated' storage layer.
  • Define how the pipeline handles malformed data and duplicates.
  • Explain how the system ensures data lineage and compliance (HIPAA).
  • Describe the strategy for partitioning the processing to isolate faults per partner.

Scale & Constraints

  • Ingest 5,000 records/sec on average, with peaks up to 20,000 records/sec.
  • Latency: E2E processing time for a record should be under 5 minutes.
  • Raw files must be stored for at least 7 years; processed records are updated daily.

Hints

Hint 1

Consider an event-driven architecture using a message broker to decouple ingestion from long-running processing tasks.

Hint 2

Use a 'Dead Letter Queue' (DLQ) pattern to isolate malformed records without halting the entire stream.

Hint 3

Implement an idempotent processing layer using a unique key derived from provider ID, patient ID, and timestamp to handle duplicate records.

Follow-up questions

  • How would you handle a schema evolution where a partner changes their data format without warning?
  • What specific metrics and monitoring would you implement to detect silent data corruption?
  • If you needed to perform cross-partner record matching (entity resolution), where would you insert this step in the pipeline?