Katkat
Back

Build a Self-Service Market Data Discovery and Support Layer

Jain GlobalData Engineer · Technical Screen
MediumData EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This problem tests your ability to design a robust metadata-driven orchestration layer that manages data lineage, access control, and quality monitoring for financial market datasets.

The Problem

Financial analysts at Jain Global ingest data from multiple vendors (Bloomberg, Refinitiv, FactSet). Your task is to design a 'Market Data Discovery and Support Layer' that allows internal users to discover available datasets, understand their lineage, and access them programmatically or via SQL/Excel interfaces.

The system must handle the following requirements:

  1. Data Discovery & Cataloging: Analysts must be able to search for tickers, financial metrics, and vendor sources.
  2. Lineage & Quality: The system must track data provenance (which vendor provided the data, when it was ingested, and any transformations applied) to ensure reproducibility in backtesting.
  3. Entitlements & Licensing: Certain datasets carry strict licensing restrictions. You must design a mechanism to enforce access controls based on user roles and vendor agreements.
  4. Data Reliability: Implement a mechanism to alert users if a data feed is stale, missing, or deviates significantly from a historical moving average (outlier detection).

Constraints & Considerations

  • Scalability: Assume the system tracks thousands of symbols across hundreds of daily feeds.
  • Interface: The system should serve as a wrapper or middleware. Think about how a Python-based quantitative analyst and an Excel-based portfolio manager would interact with this layer.
  • Design Requirements:
    • Define the metadata schema (what information is stored about a dataset?).
    • Propose the high-level architecture (services, databases, and APIs).
    • Explain the flow of data from ingestion (vendor API) to the end-user.
    • Describe how to handle 'look-ahead bias' for point-in-time data requests.

Hints

Hint 1

Focus on decoupling the user interface from the raw vendor storage by using a metadata catalog as the 'single source of truth'.

Hint 2

Use a time-versioned database schema (e.g., valid-from and valid-to timestamps) to ensure that backtesting queries always retrieve the data as it existed at the time of the trade.

Hint 3

Implement an entitlement service that validates user claims against an ACL (Access Control List) linked to specific data attributes or vendors before serving a request.

Follow-up questions

  • How would you handle a situation where a vendor retrospectively corrects historical data? How does that impact existing backtests?
  • If an analyst reports that their backtest results suddenly changed, what metrics or logs would you examine to determine if the issue is a code error or a data update?
  • How would you integrate an automated 'Quality Gate' that prevents downstream models from using data if it fails basic schema or range validation checks?