Katkat
Back

Clean OCR data and build an LLM dataset

MicrosoftMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteMachine Learning

Last updated

This problem tests the ability to design an automated pipeline for cleaning noisy OCR text output and preparing it for supervised fine-tuning of an LLM.

Scenario

You are working on a document digitalization project at Microsoft. You have a raw dataset consisting of 100,000 document pages processed through an OCR engine. The OCR output is frequently noisy, containing character substitution errors (e.g., 'l' vs '1'), broken line breaks, and hallucinations of non-existent characters in low-contrast areas. You need to prepare a high-quality dataset to fine-tune a small LLM to act as an 'OCR Post-Correction' model.

The Task

Design a systematic approach to clean this data and generate a training corpus. Your response should address the following:

  1. Data Cleaning Pipeline: Define a set of heuristic rules or statistical methods to identify "low-confidence" OCR segments. How would you handle common OCR artifacts like repeated characters or unnatural word splits?
  2. Synthetic Data Generation: Since you do not have ground truth for most of the 100k pages, describe how you would use a larger, more powerful LLM (e.g., GPT-4) to create a smaller, high-quality gold-standard dataset for fine-tuning your target model.
  3. Quality Filtering: Propose a metric or classifier to filter out documents that are too noisy to be corrected even by a sophisticated model (e.g., images that are too blurry or distorted).

Example

Input: "The 1nvestment was 500,000 dol1ars in 202O." Expected Output: "The investment was 500,000 dollars in 2020."

Constraints

  • Assume your fine-tuning model has a context window of 4,096 tokens.
  • Consider the cost of using proprietary LLM APIs for the entire dataset versus subset sampling.
  • Your solution must be scalable to multi-terabyte document repositories.

Hints

Hint 1

Consider using character-level n-gram analysis or spell-check libraries to create an 'error density' score for each document.

Hint 2

Think about 'distilling' knowledge: use a large model to correct a subset of your data and use that as the training signal for your smaller student model.

Hint 3

Look into evaluating the 'correctness' of your cleaned text by comparing the semantic embedding similarity of the OCR output against a reference database if available, or by using a secondary classifier to detect gibberish.

Follow-up questions

  • How would you handle documents with complex layouts, such as tables or multi-column text, during the cleaning phase?
  • If the OCR engine produces confidence scores, how would you incorporate them into the training objective?
  • How would you evaluate the final performance of your fine-tuned model without manual inspection of all 100k pages?