Katkat
Back

Test if social users are more engaged

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design an analytical framework to measure user engagement, account for selection bias in non-randomized groups, and define robust metrics for comparing heterogeneous user populations.

Scenario

Meta is interested in understanding whether users who actively use 'Social Features' (e.g., Groups, Events, Marketplace messaging) show higher engagement on the platform compared to users who do not use these features. You are given access to two tables:

user_activity (user_id, date, platform_event_type, time_spent_seconds) user_attributes (user_id, join_date, region, device_type)

An 'Active Social User' is defined as a user who has triggered at least one 'Social Feature' event in the last 30 days. You are tasked with determining if these users are fundamentally more engaged than the rest of the user base.

Your Task

  1. Propose a definition for 'engagement' that is robust to differences in user tenure and device type.
  2. Explain how you would account for the selection bias where highly active users might naturally gravitate toward social features, rather than the features causing the engagement.
  3. Design a metric to compare the two groups and describe the statistical method you would use to validate if the difference in engagement is significant.

Considerations

  • How do you handle the 'New User' effect where recent sign-ups have naturally different behavior than legacy users?
  • What confounding variables could skew your results, and how would you control for them (e.g., via matching or weighting)?
  • Since this is an observational study, how do you phrase your conclusion without implying strict causality?

Hints

Hint 1

Consider defining engagement as a normalized metric, such as 'daily time spent' or 'event frequency' averaged over the user's active days, rather than a raw aggregate.

Hint 2

Think about using Propensity Score Matching (PSM) to create a control group that mimics the demographic and tenure distribution of your 'Active Social User' group.

Hint 3

Consider whether you should use a t-test, or if the distribution of engagement data (which is often long-tailed) requires a non-parametric test like the Mann-Whitney U test.

Follow-up questions

  • If you discovered that 'Social Feature' usage is highly correlated with desktop usage, how would you re-adjust your model to isolate the effect of the feature itself?
  • How would you identify if there is a 'plateau effect' where increased social usage eventually leads to diminishing returns in total time spent?
  • If we were to run a randomized experiment (A/B test) to validate this, what would be the primary challenge given that social features are inherently network-dependent?