Katkat
Back

Design Data Quality and Observability Pipeline

Point72Data Engineer · Technical Screen
HardData EngineerTechnical ScreenSystem Design

Last updated

This question evaluates proficiency in designing production data orchestration, data quality validation, and observability frameworks for both batch and streaming pipelines, covering dependency management, schema and integrity checks, and operational handling of anomalous records.

Design a production data orchestration, data quality, and observability framework for batch and streaming data pipelines.

Your design should address:

  1. How would you orchestrate data pipelines with upstream and downstream dependencies?
  2. What data quality checks would you implement, such as threshold checks, upstream dependency checks, schema checks, null checks, and primary-key checks?
  3. What would a data observability framework include?
  4. If a small percentage of streaming records, for example 2%, fails validation, would you stop the entire pipeline? Explain your decision process and handling strategy.