Filter Bad Human Annotations

Scenario

You are leading the data curation pipeline for a critical LLM safety evaluation and reward modeling initiative. You have collected 500,000 pairwise preference annotations (response_A vs. response_B for a given prompt) from over 2,000 third-party crowd workers.

Preliminary audits indicate that a noticeable fraction of annotations are low quality due to bot farms, rushed workers ("click-through spammers"), and annotator confusion on complex policy guidelines. However, you also observe that some prompts are inherently subjective or ambiguous, where even diligent human annotators legitimately disagree.

Tasks

Design an end-to-end methodology to identify, filter, or re-weight noisy annotations while preserving legitimate diversity in human preferences.

1. Annotator and Task Modeling

• Propose a formal probabilistic or statistical framework to jointly model annotator reliability (skill/bias) and instance difficulty/ambiguity (e.g., extensions of Bradley-Terry, Dawid-Skene, or Item Response Theory).
• How do you separate an annotator with poor reading comprehension from an inherently hard or subjective prompt?

2. Filtering and Aggregation Pipeline

• Detail the filtering heuristic or selection rule you would use. Should noisy samples be discarded entirely, down-weighted, or sent for re-annotation?
• How do you avoid "majority-rules" collapse that filters out valid minority viewpoints on sensitive or culturally dependent prompts?

3. Verification and Downstream Impact

• Because true ground-truth labels do not exist for the full dataset, how do you construct a reliable validation benchmark to audit your filtering pipeline?
• How would you measure whether cleaning the dataset actually improves downstream reward model generalization and policy alignment, rather than simply reducing sample size?