How detect duplicate card records?
Two SigmaData Scientist · Technical Screen
Last updated
This question evaluates a candidate's competency in duplicate detection, record linkage, entity resolution, and data quality assessment within credit card transaction datasets.
You are given a dataset of credit card transaction records and suspect that some records are duplicates.
Discuss:
- What real-world situations could create duplicate-looking records.
- How you would define a true duplicate, especially when two records are very similar but not exactly identical.
- How you would detect, quantify, and extract duplicate records from the dataset in a robust way.
Your answer should consider both exact duplicates and near-duplicates caused by ingestion issues, retries, formatting differences, or multiple processing stages.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.