Katkat
Back

When use LLMs for reporting?

NatooraData Analyst · Technical Screen
MediumData AnalystTechnical ScreenMachine Learning

Last updated

This question assesses the architectural trade-offs, reliability concerns, and suitability of using LLMs for automated business reporting tasks compared to traditional deterministic methods.

Scenario

You are a Data Analyst at Natoora, a company that sources and distributes high-quality produce. Currently, your team generates daily supply-chain reports using SQL queries and BI dashboards. Management wants to explore integrating an LLM to generate 'natural language insights' based on these structured datasets to help operational managers make faster decisions.

The Task

Evaluate how you would design a system to integrate an LLM into the existing reporting pipeline. Specifically, address the following:

  1. Use-Case Fit: Identify one specific scenario in a supply chain workflow where an LLM is superior to a static dashboard, and one where it is inferior or risky.
  2. Architecture: Describe the high-level workflow for generating these reports, including how you ensure the LLM stays grounded in factual database records.
  3. Evaluation: Propose a methodology to validate the accuracy and 'truthfulness' of the LLM-generated summaries before they reach the stakeholders.

Constraints

  • The reporting involves sensitive inventory and pricing data.
  • The system must be scalable enough for hundreds of operational managers.
  • Operational managers prioritize precision over creativity.

Hints

Hint 1

Consider the difference between RAG (Retrieval-Augmented Generation) and fine-tuning for ensuring the LLM does not hallucinate inventory numbers.

Hint 2

Think about the 'Chain-of-Thought' or 'SQL-to-text' approach where the LLM is given SQL query results rather than direct access to the raw database.

Hint 3

Focus on how you would handle non-deterministic output: how do you automate the checking of an LLM response against the hard values in your source data?

Follow-up questions

  • How would you handle cases where the LLM's interpretation of a data trend contradicts the historical trend identified by a deterministic statistical model?
  • What specific metrics would you track to monitor for 'drift' or degradation in the quality of the insights over time?
  • If the operational managers report that the LLM's tone is too robotic or inconsistent, how would you address that while maintaining data integrity?