Design a Resumable, Parallel Fetcher for Huge Dataset Releases

Your team publishes large dataset releases: a release is a named, versioned bundle of files (for example [email protected]) of 50 GB to 3 TB, split across a few to several thousand files of 100 MB to 20 GB each. Hundreds of compute nodes need to pull a release onto local NVMe disk before a job starts. Downloads go through an HTTP object store or CDN that supports Range requests.

Design the client-side fetcher: a daemon or library that runs on each node, downloads a release in parallel, survives crashes and network failures without starting over, and only ever exposes a release to jobs once the whole release is present and verified.

Functional requirements

1. fetch(release_id) -> local_path blocks (or reports progress) until the release is ready, then returns a directory that jobs can read.
2. Download files concurrently and split large files into byte ranges fetched in parallel.
3. If the process crashes, the node reboots, or a connection drops, a later fetch reuses every piece that was already verified.
4. A job must never see a half-written file or a mix of files from two different releases.
5. Several jobs on the same node may request the same release at the same time. It should be downloaded once.

Non-functional requirements

• Aim to saturate the node's link (assume 25 Gbps) without exhausting memory (budget about 1 GB for buffers), file descriptors, or the origin's rate limits.
• Integrity: every byte must be verified against a checksum the publisher provides.
• The storage overhead on the local disk is bounded. Do not keep two full copies of the release.
• Releases are immutable once published, but the store can contain a newer object at the same URL after a republish, so make sure you never combine bytes from two object versions.

Deliverables

• Part 1: planning and transfer. The release manifest format, how you turn it into a piece plan, how concurrency is scheduled and bounded, and what you validate on every HTTP response.
• Part 2: resume and publish. What progress state you persist and where, how a restarted fetcher decides which pieces it can trust, and the exact steps that turn a staging area into a visible, complete release.
• Part 3: scale and trade-offs. What changes when 500 nodes pull the same release at once, and how you would tune piece size and parallelism.

Questions worth asking first: Are checksums published per file, per piece, or both? Does the store return a version identifier (ETag or generation number) per object? Do jobs need the whole release to be visible at once, or can they stream individual files?