Katkat
Back

Evaluate Noisy Data for LLM Post-Training

MercorMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Design a curation and validation framework to evaluate, filter, and safely integrate a large, noisy dataset into an LLM post-training pipeline without degrading model performance.

Scenario

You are a Machine Learning Engineer responsible for post-training (Supervised Fine-Tuning and Direct Preference Optimization) a 7B parameter reasoning and instruction-following model. Your current baseline model was trained on an internal, human-verified "gold" dataset of 40,000 instruction-response pairs.

A partner team provides an external, weakly-supervised dataset of 500,000 prompt-response pairs scraped from developer forums, user submissions, and synthetic task generations. While the dataset promises broader domain coverage, preliminary inspection reveals significant noise: hallucinated reasoning steps, broken code snippets, inconsistent tone, potential PII (e.g., API keys, email addresses), and toxic content.

Task

Design an end-to-end evaluation and filtering strategy to determine whether and how to incorporate this data into your post-training pipeline. Your proposed approach should address the following:

  1. Data Auditing and Automated Filtering:

    • Define automated heuristics (e.g., length ratios, perplexity filtering via reference models, exact/near deduplication) to discard low-quality samples quickly.
    • Detail a model-based scoring strategy (e.g., reward models, LLM-as-a-judge, self-consistency checks) to evaluate the factual correctness and step-by-step reasoning quality.
    • Specify checks for safety, toxicity, and PII compliance.

  2. Distributional Fit and Diversity:

    • How will you measure whether this dataset complements your gold data rather than distorting your intended target distribution?
    • Propose methods (such as embedding clustering or k-nearest neighbors to gold validation prompts) to identify high-density or underrepresented task clusters.

  3. Empirical Validation and Regression Prevention:

    • Outline a controlled ablation study to test different mixture ratios of gold data to curated external data.
    • Define a suite of evaluation benchmarks to detect capability degradation (e.g., catastrophic forgetting on core reasoning benchmarks, style drift, alignment tax, or safety regressions).
    • Establish explicit go/no-go criteria before promoting a newly trained model checkpoint to production.

Hints

Hint 1

Differentiate between low-cost rule-based filters (like regex for PII, MinHash for deduplication, and length/perplexity thresholds) and high-cost model-based judges for deep reasoning checks.

Hint 2

Use semantic embeddings to map the external prompts against your target distribution, ensuring you only sample instances that fill actual coverage gaps.

Hint 3

Formulate a regression testing suite that tests not only targeted domain tasks but also fixed anchor benchmarks for general reasoning, safety, and instruction-following consistency.

Follow-up questions

  • How would you address instances where a synthetic reasoning trace produces the correct final answer despite flawed intermediate deductions?
  • If incorporating a high-scoring slice of data causes a drop in safety or general benchmark performance, what diagnostic methods (e.g., gradient influence functions, data attribution) would you use to locate the problematic samples?
  • How do you adjust sampling weights or curriculum schedules during SFT if the filtered external data has a significantly different average token length than your gold dataset?