Katkat
Back

Select high-quality math documents from crawls

OpenAIMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteML System Design

Last updated

This question assesses the ability to design a scalable pipeline that ingests, cleans, extracts, and filters high-quality mathematical content from diverse, noisy web-crawled sources.

Problem Statement

You are tasked with building a production-grade system to ingest petabyte-scale web crawls (HTML, PDF, and images) and curate a high-quality dataset of mathematical content (theorems, proofs, and derivations) for training LLMs. The raw data is highly noisy, containing duplicates, non-math text, and low-quality OCR results.

Functional Requirements

  1. Extraction: Normalize content across heterogeneous formats.
  2. Quality Scoring: Identify and prioritize high-value mathematical content.
  3. Safety & Deduplication: Remove copyrighted material and near-duplicate content.

Scale and Constraints

  • Scale: 100TB of raw documents per day.
  • Latency: The pipeline is batch-oriented but must process daily throughput efficiently.
  • Precision: The final dataset must have a high signal-to-noise ratio of mathematical rigor.

Expectations

Design a system architecture that covers:

  • Data Pipeline: Steps from raw crawl to a clean training-ready corpus.
  • Modeling: Approaches for document classification, OCR, and quality scoring.
  • Storage/Indexing: How to manage document versions and metadata.
  • Monitoring: How to track data drift and ensure training set quality over time.

Hints

Hint 1

Consider using a multi-stage approach where you first filter documents by language and mathematical symbol density before applying expensive OCR or LLM-based scoring.

Hint 2

Think about using hashing techniques (e.g., MinHash or SimHash) for deduplication at scale to avoid redundant processing.

Hint 3

Discuss the trade-off between using lightweight heuristic-based scoring (e.g., LaTeX-to-text ratio) versus heavy-weight model-based scoring.

Follow-up questions

  • How would you handle documents where the OCR fails to interpret complex mathematical notation correctly?
  • How do you implement a feedback loop to refine your quality-scoring model based on manual human review?
  • If you discovered your dataset had a bias toward a specific sub-field of mathematics, how would you adjust the pipeline to re-balance the training data?