Katkat
Back

Calibrate LLM output to match Word formatting

MicrosoftMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to design an end-to-end system for enforcing strict structured formatting (e.g., Docx/Markdown) on LLM-generated text, focusing on constraint enforcement and reliability.

Problem Statement

Microsoft Word documents often require strict hierarchical structures (headers, tables, numbered lists) and styling. You are tasked with designing a system that takes a natural language prompt and generates content that must conform to a predefined schema (e.g., a specific JSON-based document structure that maps to Word formatting).

Requirements

  • Functional: The system must produce output that is guaranteed to parse correctly according to the target schema. If the LLM generates malformed output, the system must either prevent it or repair it.
  • Performance: Users expect low latency; document generation should take seconds, not minutes.
  • Reliability: The system must handle long-context documents without losing format consistency.

Design Components

  1. Representation: How do you define the document schema so that the LLM understands the formatting constraints (e.g., constraints on table cell types or list depth)?
  2. Constraint Enforcement: Propose a method to ensure the model follows these constraints (e.g., grammar-based decoding, prompt engineering, or post-processing).
  3. Repair Strategy: If the model violates the schema, how do you handle it without a full regeneration?
  4. Scalability: How do you ensure the system remains performant for large documents (e.g., 50+ pages)?

Expectations

  • Provide a high-level architecture diagram describing the interaction between the LLM and the validation layer.
  • Discuss the trade-offs between 'soft' constraints (prompting) and 'hard' constraints (logit bias/constrained decoding).
  • Define metrics for evaluating the system (e.g., Schema Compliance Rate, Token Efficiency).

Hints

Hint 1

Consider using formal grammars like BNF or JSON schemas to constrain the output space of the LLM.

Hint 2

Think about whether it is more efficient to validate tokens in real-time during inference versus using an LLM-based agent to rewrite malformed outputs.

Hint 3

For long documents, consider a chunk-based generation approach and discuss how to maintain global formatting consistency across chunk boundaries.

Follow-up questions

  • How would you handle a scenario where the user's natural language request inherently conflicts with the provided document schema?
  • If you have to support real-time collaboration features (like tracking changes in Word), how does that influence your data model design?
  • How would you measure the cost-benefit of adding a 'repair' agent versus simply forcing the model to generate the correct format via constrained decoding?