Katkat

Implement a Duplicate-File Reporter

AnthropicSoftware Engineer · Technical Screen

Last updated

Implement an efficient file system scanner that identifies duplicate files while minimizing disk I/O and gracefully handling filesystem errors.

Task

Implement a function find_duplicates that scans a filesystem directory recursively and returns groups of duplicate files based on their content.

Reading entire files from disk is expensive. An optimal implementation balances CPU and disk I/O by filtering candidate duplicates in stages rather than hashing every file completely. Additionally, real-world filesystems can have broken symlinks, permission issues, or files modified during traversal.

Requirements

  1. Return a list of lists, where each inner list contains paths of files that have identical byte content. Omit unique files (every group must have at least 2 paths).
  2. Minimize I/O: Only read content when two or more files share the same file size.
  3. Robustness: Skip files that cannot be read (e.g., PermissionError, FileNotFoundError, broken symlinks) without crashing.
  4. Zero-byte files: Zero-byte files should be treated as duplicates of each other if there are two or more.

Function Signature

def find_duplicates(root_path: str) -> list[list[str]]:
    """
    Recursively scans root_path and returns groups of duplicate file paths.
    Each group contains 2 or more paths with identical content.
    """
    pass

Example

Suppose the directory /data has the following files:

/data/a.txt      (size: 100 bytes, content: "hello world...")
/data/b.txt      (size: 100 bytes, content: "hello world...")
/data/c.txt      (size: 100 bytes, content: "different!...")
/data/d.txt      (size: 50 bytes,  content: "short text")
/data/sub/e.txt  (size: 100 bytes, content: "hello world...")

Output:

[
    ["/data/a.txt", "/data/b.txt", "/data/sub/e.txt"]
]

Explanation: c.txt has the same size (100 bytes) but different content. d.txt is unique in size. Only a.txt, b.txt, and e.txt share identical content.

Constraints

  • Directory depth can be arbitrarily nested.
  • File sizes can range from 0 bytes to tens of gigabytes.
  • Paths should be returned as strings (the order of groups and files within groups does not matter).

Hints

Hint 1

Start by grouping files by their file size using metadata inspection; files with unique sizes can never be duplicates and need never be read.

Hint 2

For files with matching sizes, compare a small prefix (e.g., first 1 KB to 4 KB) using a fast hash before reading and hashing the entire file.

Hint 3

Wrap disk read operations in `try-except` blocks to handle unreadable files, broken symlinks, or files deleted mid-traversal without aborting the scan.

Follow-up questions

  • How would you handle hard links so that the same inode referenced via multiple directory entries is not mistakenly reported as duplicate file content?
  • How would you parallelize this pipeline across multiple CPU cores without causing disk thrashing on HDDs vs SSDs?
  • What strategy would you use to eliminate the possibility of hash collisions if cryptographic certainty is required for mission-critical deletion?