Katkat
Back

North-star and guardrail metrics for generative AI across Meta's apps

MetaProduct Manager · Other
MediumProduct ManagerOtherAnalytics & Experimentation

Last updated

A product manager metrics question on judging whether Meta's generative AI features, such as its in-app assistant, creation tools and ad tools, are succeeding. It tests choosing a north-star metric that reflects real value, supporting and guardrail metrics, and how to show the AI actually caused any change.

Meta has added generative AI to many of its products: an assistant that people can chat with inside its apps, tools that create or edit images and video, and features that help businesses produce ads. As a product manager at Meta, explain how you would measure whether these AI features are succeeding.

Clarifying Questions

  • Should the answer cover the whole set of AI features, or focus on one, such as the assistant, the image and video creation tools, or the tools for advertisers?
  • Whose success matters most here: people using Meta's apps, creators and businesses, or Meta's own engagement and ad revenue?
  • Is this for an ongoing health dashboard, or to decide whether a particular AI feature keeps getting investment?

Hints

Hint 1

Usage is not the same as value: AI entry points sit in places people already tap, so raw usage can climb without anyone being helped. Look for signals that show a person got something useful and chose to come back.

Follow-up questions

  • Daily assistant users are rising, but most sessions are a single question typed into the search bar. Is that success, and how would you tell?
  • How would you measure whether AI-generated images make the feed better or worse for viewers who never use the creation tools?
  • As usage grows, the cost of serving responses climbs sharply. How would you decide whether a feature still pays for itself?