Katkat
Back

Collect labels without existing data

ShopifyMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteAnalytics & Experimentation

Last updated

This question evaluates a machine learning engineer's competency in label definition and leakage prevention, instrumentation and logging design, programmatic and human-in-the-loop labeling pipelines, experimental approaches for eliciting outcomes, bias mitigation, and continuous label quality monitoring.

You must build a model but have no labeled data. How would you define the label, collect or create it ethically and at scale, and validate quality? Discuss instrumentation and logging schemas, heuristic/weak supervision and programmatic labeling, human-in-the-loop pipelines, proxy labels, controlled experiments or exploration to elicit outcomes, sampling strategies to reduce bias, gold sets and inter-rater agreement, and continuous data quality monitoring.