Katkat
Back

Design quality checks for spreadsheet LLM data

MicrosoftMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenML System Design

Last updated

This problem tests the ability to design an end-to-end data validation pipeline for spreadsheet data and determine the necessity of fine-tuning a Large Language Model (LLM) for specific structured data tasks.

Scenario

You are building an automated pipeline that processes unstructured user inputs and translates them into structured spreadsheet updates (e.g., cell formulas or data population). The inputs come from a diverse set of users, leading to inconsistent formatting, missing headers, and semantic ambiguities. You have access to a general-purpose LLM via API.

Task

Design a system to validate the quality of the LLM's output before it is committed to a production spreadsheet. Your design should address:

  1. Data Quality Validation: How do you enforce schema constraints (e.g., data types) and semantic consistency (e.g., logical checks between columns) on the LLM's raw output?
  2. Evaluation Strategy: How would you build a ground-truth dataset for evaluating whether the model needs fine-tuning versus using few-shot prompting?
  3. Decision Criteria: What signals would indicate that a pretrained LLM is insufficient and that a fine-tuning phase is required?

Requirements

  • Define the architecture of your validation layer.
  • Describe the metrics you would use to compare a baseline (prompting) against a potential fine-tuned model.
  • Address potential data leakage concerns if your training data is sourced from historical user interaction logs.

Hints

Hint 1

Consider implementing a two-layer validation approach: one for structural syntax (e.g., JSON schema/CSV format) and one for domain-specific business rules.

Hint 2

Think about using 'golden sets' created via manual verification to measure precision and recall of the model's structure-extraction capabilities.

Hint 3

Discuss the trade-off between the overhead of hosting a fine-tuned model and the cost-per-token/latency benefits of using a smaller, distilled model versus a large general-purpose LLM.

Follow-up questions

  • How would you handle cases where the LLM produces a syntactically correct formula but one that references incorrect spreadsheet ranges?
  • If you decide to fine-tune, what specific strategies would you use to prevent 'catastrophic forgetting' of general reasoning capabilities?
  • How does your design change if the input data involves sensitive PII (Personally Identifiable Information) that cannot leave the local environment?