Katkat
Back

Clarify an Ambiguous Product Question and Define a Metric

FigmaData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to transform an abstract product objective into a rigorous measurement framework, focusing on metric selection and experimental design.

The Prompt

Figma is considering introducing a new 'AI-Assisted Layer Naming' feature that automatically suggests intuitive names for new layers based on their content. The product lead asks: 'How should we evaluate if this feature is making our designers more productive?'

Your Task

Develop a measurement framework to evaluate this feature. Your answer must cover the following:

  1. Clarification: Identify 2–3 questions you would ask the product lead to resolve ambiguity in the term 'productive'.
  2. Metric Definition: Propose a primary metric. Define the unit of analysis, the population (e.g., all users vs. active power users), and the time horizon.
  3. Guardrails: Define at least two guardrail metrics to ensure that the AI isn't negatively impacting the user experience or system performance.
  4. Design: Describe a high-level experimental design to determine the impact of the feature, including how you would handle potential network effects or bias (e.g., if users who use AI features are inherently more tech-savvy).

Constraints & Considerations

  • Assume you have access to event logs (layer creation, rename events, AI suggestion triggers, and 'accept/reject' actions).
  • Distinguish between 'automated efficiency' and 'creative workflow quality'.
  • Think about the risk of 'over-reliance'—what happens if users stop thinking about their layer structure entirely?

Hints

Hint 1

Consider that 'productivity' could mean reducing the time spent on manual tasks, but it could also mean increasing the frequency of design iterations.

Hint 2

For guardrail metrics, consider the 'undo' rate or the rate at which users manually rename a layer immediately after accepting an AI suggestion.

Hint 3

Think about the difference between a user-level metric (e.g., time per file) and an event-level metric (e.g., rename latency).

Follow-up questions

  • How would you account for the 'novelty effect' where users might engage with the feature more during the first week simply because it is new?
  • If the feature consumes significant compute resources, how would you incorporate cost-per-design-action into your decision-making framework?
  • What would you do if your primary metric shows a significant increase in speed, but guardrail metrics show a drop in 'design organization' scores as reported by user surveys?