Katkat
Back

Find A Low-Quality Annotator From Label Data

MetaMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteData Manipulation (SQL/Python)

Last updated

This problem tests your ability to clean noisy labeling datasets and design metrics to detect annotator bias or performance degradation.

Problem Statement

You are working on an image classification task for a new AI feature. You have a dataset of labels collected from multiple annotators. You need to identify a single 'low-quality' annotator who is negatively impacting the model's performance.

The dataset is provided as a pandas DataFrame df with the following columns:

  • annotator_id: Unique identifier for the worker.
  • image_id: Unique identifier for the image.
  • label: Integer (0 or 1).
  • is_ground_truth: Boolean indicating if this is a verified gold-standard label.

Task

Write a function identify_low_quality_annotator(df) that returns the annotator_id of the worker whose labels are most suspicious. Define 'suspicious' as having the lowest agreement rate with the ground truth, but only consider annotators who have labeled at least 20 ground-truth images to ensure statistical significance. If there are ties, select the one with the smallest total number of annotations.

Constraints

  • df contains 10^5 to 10^6 rows.
  • annotator_id and image_id are strings.
  • Handle cases where an annotator might not have any ground-truth labels (exclude them from calculations).

Example

import pandas as pd

# Input
data = {
    'annotator_id': ['A', 'A', 'B', 'B', 'A', 'B'],
    'label': [1, 1, 0, 1, 1, 0],
    'is_ground_truth': [True, True, True, True, True, True]
}
df = pd.DataFrame(data)

# Function call
# identify_low_quality_annotator(df) -> returns 'B' (Agreement 50% vs 'A' 100%)

Edge Cases

  • Annotators with zero ground-truth labels.
  • Scenarios where no annotator meets the minimum 20-label threshold.
  • Datasets with extreme class imbalance.

Hints

Hint 1

First, filter the DataFrame to keep only rows where `is_ground_truth` is True.

Hint 2

Group the filtered data by `annotator_id` and calculate the mean of the equality condition `df['label'] == df['ground_truth_label']` (you will need to join the ground truth labels to the annotator labels first).

Hint 3

Use a mask or query to filter out annotators who have labeled fewer than 20 images before sorting by their performance metric.

Follow-up questions

  • How would you handle a scenario where 'ground truth' is unavailable and you must rely on inter-annotator agreement (e.g., Cohen's Kappa) instead?
  • If an annotator is identified as low-quality, how would you programmatically 'clean' the dataset without simply deleting all their contributions?
  • How does class imbalance in the ground-truth labels affect your metric, and how would you adjust the scoring to prevent bias toward the majority class?