Katkat
Back

Choose a North Star Metric for a Malicious-Webpage Classifier

GoogleData Analyst · Onsite
HardData AnalystOnsiteAnalytics & Experimentation

Last updated

Choose and measure a malicious-webpage classifier metric using exposure-weighted recall, false-warning guardrails, and unbiased label audits.

Choose a North Star Metric for a Malicious-Webpage Classifier

A classifier flags whether a webpage a user is browsing is malicious. Propose a north star metric for the classifier and explain how you would measure it. Include the guardrails needed to prevent a higher metric from masking harm to users. Clarify the unit of evaluation and how reliable labels would be obtained.

Hints

Hint 1

Count opportunities for harm: A rare malicious page visited often may matter differently from many pages that no one visits.

Follow-up questions

  • Why can classifier accuracy improve while malicious exposure gets worse?
  • What additional evidence is needed to claim that warnings prevented harm?