Implement file deduplication at scale
AnthropicSoftware Engineer · Onsite
Last updated
Implement streaming file deduplication that filters by size, fingerprint and hash, then compares bytes to rule out hash collisions. It covers large-file chunking, batched disk I/O, parallelizing across cores or machines, a dry-run report, and hard links that never cross devices. Follow-ups turn it into a command-line tool that handles symbolic links, unreadable files, error cases, and options to move or delete duplicates.
Write a program to deduplicate files in a very large directory tree. Identify groups of identical files without loading entire files into memory. Outline your approach to hashing (e.g., size filter, partial hash, full hash), chunking for very large files, and handling hash collisions. Support a mode that replaces duplicates with hard links (when safe) and a dry-run report of duplicate sets. Explain time and space complexity, how you batch disk I/O, and how you would parallelize across CPU cores or machines.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.