Katkat
Back

Choose Components for a Data Warehouse

ByteDanceData Engineer · HR Screen
MediumData EngineerHR ScreenSoftware Engineering Fundamentals

Last updated

This question tests the ability to distinguish between distributed data processing frameworks, storage engines, and relational databases, evaluating your understanding of trade-offs in large-scale data infrastructure.

The Scenario

You are tasked with selecting a technology stack for a new data warehouse project at a high-growth company. The requirements are as follows:

  1. Storage of 500TB of semi-structured log data with periodic batch ETL jobs.
  2. Support for ad-hoc SQL queries by analysts on structured data.
  3. A requirement for low-latency dashboards for executive reporting.
  4. Strict data governance and schema enforcement.

The Task

Compare and contrast the following technologies in the context of the requirements above: Apache Hive, Apache Spark, Hadoop (HDFS), and SQL Server. Specifically, explain which combinations you would use to build an end-to-end pipeline and justify why you would exclude any of the listed technologies for specific layers (storage, compute, or serving).

Hints

Hint 1

Consider the separation of compute and storage: HDFS is fundamentally a distributed file system, while the others are compute engines or database management systems.

Hint 2

Think about the difference between OLAP (Online Analytical Processing) workloads, which favor throughput, and the low-latency requirements of standard operational SQL Server environments.

Hint 3

Evaluate governance capabilities, such as ACID compliance and schema-on-read versus schema-on-write architectures.

Follow-up questions

  • How would you handle the evolution of data schemas if the log formats change frequently?
  • If the query latency requirements shifted from minutes to sub-second responses, how would you change your storage layer?
  • What are the specific trade-offs between using Spark SQL for ELT versus a traditional ETL process in a structured database?