Implement a Duplicate-File Reporter
AnthropicSoftware Engineer · Technical ScreenLast updated
Implement an efficient file system scanner that identifies duplicate files while minimizing disk I/O and gracefully handling filesystem errors.
Task
Implement a function find_duplicates that scans a filesystem directory recursively and returns groups of duplicate files based on their content.
Reading entire files from disk is expensive. An optimal implementation balances CPU and disk I/O by filtering candidate duplicates in stages rather than hashing every file completely. Additionally, real-world filesystems can have broken symlinks, permission issues, or files modified during traversal.
Requirements
- Return a list of lists, where each inner list contains paths of files that have identical byte content. Omit unique files (every group must have at least 2 paths).
- Minimize I/O: Only read content when two or more files share the same file size.
- Robustness: Skip files that cannot be read (e.g.,
PermissionError,FileNotFoundError, broken symlinks) without crashing. - Zero-byte files: Zero-byte files should be treated as duplicates of each other if there are two or more.
Function Signature
def find_duplicates(root_path: str) -> list[list[str]]:
"""
Recursively scans root_path and returns groups of duplicate file paths.
Each group contains 2 or more paths with identical content.
"""
pass
Example
Suppose the directory /data has the following files:
/data/a.txt (size: 100 bytes, content: "hello world...")
/data/b.txt (size: 100 bytes, content: "hello world...")
/data/c.txt (size: 100 bytes, content: "different!...")
/data/d.txt (size: 50 bytes, content: "short text")
/data/sub/e.txt (size: 100 bytes, content: "hello world...")
Output:
[
["/data/a.txt", "/data/b.txt", "/data/sub/e.txt"]
]
Explanation: c.txt has the same size (100 bytes) but different content. d.txt is unique in size. Only a.txt, b.txt, and e.txt share identical content.
Constraints
- Directory depth can be arbitrarily nested.
- File sizes can range from
0bytes to tens of gigabytes. - Paths should be returned as strings (the order of groups and files within groups does not matter).
Hints
Hint 1
Start by grouping files by their file size using metadata inspection; files with unique sizes can never be duplicates and need never be read.
Hint 2
For files with matching sizes, compare a small prefix (e.g., first 1 KB to 4 KB) using a fast hash before reading and hashing the entire file.
Hint 3
Wrap disk read operations in `try-except` blocks to handle unreadable files, broken symlinks, or files deleted mid-traversal without aborting the scan.
Follow-up questions
- How would you handle hard links so that the same inode referenced via multiple directory entries is not mistakenly reported as duplicate file content?
- How would you parallelize this pipeline across multiple CPU cores without causing disk thrashing on HDDs vs SSDs?
- What strategy would you use to eliminate the possibility of hash collisions if cryptographic certainty is required for mission-critical deletion?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.