Evaluating the Impact of Duplicate and Stolen Posts on a Content Platform

Evaluating the Impact of Duplicate and Stolen Posts on a Content Platform

You are a data scientist at a large user-generated-content platform (think a social feed where users publish posts and others engage with them). Product leadership is worried about duplicate posts (the same content posted more than once, sometimes by the same author, sometimes re-uploaded by others) and stolen posts (one user re-publishing another user's original content as their own, without credit).

The platform is considering shipping detection-and-enforcement systems (deduplication, "stolen content" takedowns, attribution back to the original author). Your job is to design how the company would measure the impact of duplicate and stolen posts — and of any intervention against them — on the health of the platform.

This is a multi-part case. Work through each part in order.

Constraints & Assumptions

• The platform has tens of millions of daily active users and millions of posts created per day; content is a mix of text and media.
• You can instrument arbitrary client and server events and run randomized experiments.
• A content-similarity / near-duplicate detection service exists or can be built (hashing + embeddings); it returns a similarity score, not ground truth.
• "Impact" is ultimately about long-term ecosystem health (original-creator supply and consumer retention), not a single short-term click metric.

Clarifying Questions to Ask

• What is the platform's primary business goal here — protecting original creators (supply side), improving consumer experience (demand side), or legal/compliance risk from stolen IP? The metric hierarchy changes depending on which.
• Are duplicate posts and stolen posts being treated as the same problem or two different problems? (Self-duplication vs cross-author theft have different harms and different fixes.)
• What enforcement action is on the table — demotion in ranking, hard removal, watermark/attribution to the original author, or creator-facing warnings? The intervention determines what we can experiment on.
• What is our tolerance for false positives? Wrongly removing an original author's content is far costlier than missing one thief.
• Over what horizon does leadership want the impact measured — a 2-week experiment readout, or long-term creator retention over months?
• Do we have reliable ground-truth labels (e.g., human-reviewed cases or DMCA reports) to validate any automated "stolen" classifier against?

Part 1 — Choosing metrics

Propose the set of metrics you would use to quantify the impact of duplicate and stolen posts on the platform. Organize them so leadership understands what each one is for, and identify which single metric (or small set) you would treat as the primary success metric versus guardrails.

What This Part Should Cover

• A structured metric tree that separates prevalence/quality, creator/supply-side, and consumer/demand-side metrics, with each metric tied to a mechanism of harm.
• An explicit choice of primary metric(s) vs guardrails, with the false-positive rate on originals treated as a hard guardrail.
• Recognition that prevalence is a gameable diagnostic (threshold-dependent), not a north star, and that share-of-impressions/engagement is often more decision-relevant than raw counts.

Part 2 — Defining "stolen posts"

Write down an operational definition of a stolen post that an engineering system could actually compute at scale, then critique it: what are the failure modes of your own definition (false positives and false negatives)?

What This Part Should Cover

• An operational, computable definition combining a similarity test and an originality/ownership test (substantially-similar, posted later, different author, no attribution).
• A concrete inventory of false-positive sources (quotes, reactions/duets, memes/templates, commodity news, licensed/self reposts) and false-negative sources (paraphrase/translation, cropped/re-encoded media, sub-threshold similarity, cross-platform imports).
• Awareness that the definition is threshold-dependent and cost-asymmetric (harming an original ≫ missing a thief), motivating softer actions or human review in ambiguous cases.

Part 3 — Designing the experiment

You want to ship a "stolen-content suppression" intervention (demote or remove stolen reposts and attribute originals) and measure its causal effect. Design the experiment. The interviewer explicitly flagged that network effects complicate this — explain what the network effect is here and how it threatens a naive user-level A/B test, then propose a design that addresses it.

Clarifying Questions for this Part

• How self-contained are our communities/geos — do most views and reposts stay within a region or language, or does content cross freely? This determines whether cluster randomization actually bounds spillover.
• What effect size does leadership consider meaningful, and how long can the experiment run? Clustering inflates variance, so the minimum detectable effect (MDE) and runtime budget drive the design choice.

What This Part Should Cover

• A correct, named diagnosis: network effects = interference / SUTVA violation, with a clear mechanism for how treatment leaks into control through shared content.
• A reasoned statement of the direction of bias (contamination shrinks the estimate toward zero) and the practical consequence.
• At least one interference-robust design (cluster/community/geo, ego-network, creator/content-level, or switchback) with explicit bias-vs-variance / power trade-offs, plus pre-registration of the primary metric, an MDE/power check, and a variance-reduction plan.

Part 4 — Interpreting a metric drop

The experiment ships to the treatment arm and you observe that one of your engagement metrics dropped in treatment relative to control. Walk through how you would diagnose why — enumerate the plausible explanations (both "the intervention is genuinely bad" and "the metric is misleading") and how you'd distinguish them.

What This Part Should Cover

• A disciplined refusal to read a single metric literally, sorting hypotheses into a "good drop" bucket (removed hollow engagement; engagement reallocated to originals) and a "bad drop" bucket (over-suppression/false positives, feed-quality hole, UX/latency or instrumentation bug).
• A concrete diagnostic playbook: segment the drop, read guardrails and quality-weighted metrics, inspect the detector's false-positive rate, and check the time trend for novelty effects.
• A decision rule that ships only when primary metrics and guardrails are healthy, treating a raw-volume drop alone as non-blocking — and awareness of pitfalls like Simpson's paradox in aggregated comparisons.