Katkat
Back

Parallelize chunked file download and verify integrity

BasetenSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenCoding & Algorithms

Last updated

This problem tests your ability to implement a concurrent file downloader that fetches file segments in parallel, handles partial failures with retries, and verifies overall data integrity using a checksum.

Problem Statement

You are building a robust client to download a large file from a remote server. The file is split into N fixed-size chunks, and the server provides an API to download specific byte ranges.

Implement a function download_file(file_id, total_chunks, chunk_size, checksum) that:

  1. Downloads all chunks in parallel using a pool of worker threads or an asynchronous executor.
  2. Retries a chunk download up to 3 times if it fails.
  3. Assembles the chunks into a single file on disk in the correct order.
  4. Verifies the integrity of the final file by comparing its SHA-256 hash against the provided checksum.

Function Signature

def download_file(file_id, total_chunks, chunk_size, checksum):
    """
    - file_id: Unique identifier for the remote file.
    - total_chunks: Total number of chunks.
    - chunk_size: Size of each chunk in bytes.
    - checksum: Expected SHA-256 hex string of the final file.
    Returns True if successfully verified, False otherwise.
    """

Assumptions

  • You have a provided helper function fetch_chunk(file_id, chunk_index) that returns raw bytes for the specified chunk or raises an exception on failure.
  • The chunks must be written to disk sequentially to maintain the correct file structure (e.g., using seek() and write() on a file handle).
  • Memory limit: You cannot hold the entire file in RAM at once; process chunks one by one or in small batches.

Example

  • total_chunks = 3, chunk_size = 1024.
  • The downloader should fetch index 0, 1, and 2 concurrently.
  • Once all chunks are written to file_id.bin, calculate the hash of file_id.bin and compare.

Edge Cases

  • What happens if a chunk returns the wrong data? (Note: assume fetch_chunk implies server-side validation, but you must ensure the file is assembled correctly).
  • Ensuring thread safety when writing to the shared file descriptor.

Hints

Hint 1

Use a thread pool (like Python's ThreadPoolExecutor) to manage concurrent downloads, and a thread-safe lock or file `seek()` offsets to ensure chunks are written to the correct locations.

Hint 2

Store successfully downloaded chunks in a temporary structure or write them directly to their calculated positions in the file using random access I/O.

Hint 3

To verify the checksum efficiently without loading the whole file into RAM, read the file in small, manageable buffer blocks and pass them to the hashing function incrementally.

Follow-up questions

  • How would you modify this implementation to handle a situation where the network is highly unstable and chunks are very large?
  • What changes would be required if the server provided a manifest file indicating the checksum for each individual chunk in addition to the total file checksum?
  • How would you implement a progress bar that updates in real-time as chunks are completed?