Katkat
Back

How would you measure product success?

FigmaData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to define a comprehensive measurement framework for a B2B SaaS collaborative tool, balancing growth, engagement, and retention metrics.

The Scenario

Figma is considering introducing a new feature: 'Auto-Component Suggestion,' which uses machine learning to automatically group design elements into components for users as they work in the editor. As a Data Scientist, you are tasked with evaluating the success of this feature.

Your Task

Design a measurement framework for this feature. Your response should address:

  1. Metrics Strategy: Define a North Star metric, a set of input (leading) metrics, and at least two guardrail metrics.
  2. Unit of Analysis: Figma is a collaborative tool where teams work together. Explain how you would choose between measuring this at the individual user level versus the team/workspace level, and why.
  3. Power Users: How would you define a 'Power User' of this specific feature? How would you validate that your definition is actually correlated with long-term retention?
  4. Causality: How would you distinguish between the lift provided by the feature and existing usage trends?

Constraints

  • Focus on the 'Design' phase of the product lifecycle.
  • Assume you have access to event-level clickstream data and subscription status logs.
  • Consider the trade-off between immediate efficiency and the potential 'black box' effect on design quality.

Hints

Hint 1

Consider whether the North Star should prioritize feature adoption or the time saved per design file.

Hint 2

Think about the impact of the feature on 'Design Debt'—could auto-suggesting components lead to poor organization if not used correctly?

Hint 3

Use a cohort-based analysis approach to validate if users who engage with the feature are more likely to renew their subscription next cycle.

Follow-up questions

  • How would you account for 'network effects' where a feature's utility increases only when multiple team members adopt it?
  • If the feature shows high engagement but low user satisfaction in surveys, how would you investigate the discrepancy?
  • How would you set up an A/B test for this feature, given that collaborative editing in the same file makes independent control groups difficult?