How to stream a large file to 1000 hosts fastest

Scenario

You are tasked with designing a rapid distribution service for an AI infrastructure cluster. Before starting a distributed training or fine-tuning run, a 100 GB model checkpoint must be broadcast from a central storage node (or single host) to 1,000 compute worker hosts as fast as possible. Minimizing the makespan (the elapsed time until the last host completes the download) is critical to reducing idle GPU cluster time.

Requirements

Functional Requirements

• Distribute a single 100 GB file from one origin node to 1,000 worker nodes.
• Ensure byte-level integrity verification across all target nodes.
• Provide real-time status tracking and progress reporting across the fleet.

Non-Functional Requirements & Scale

• Scale: Exactly 1,000 target hosts located across multiple racks in a single data center.
• Network: Origin and worker hosts each have 25 Gbps network interface cards (NICs). Intra-rack bandwidth is abundant; inter-rack spine-leaf bisection bandwidth may be constrained.
• Makespan: Must get as close as possible to the theoretical minimum transmission time permitted by the physical network.
• Fault Tolerance: A small number of node dropouts, crashes, or stragglers should not stall distribution to the rest of the fleet.

Key Design Areas to Cover

1. Theoretical Lower Bound & Bottleneck Analysis: Calculate the theoretical minimum makespan under a naive client-server model versus collaborative distribution topologies (e.g., k-ary tree, ring pipelining, BitTorrent-style swarming).
2. Chunking and Pipelining: How should the file be partitioned, pipelined, and assembled on target hosts?
3. Topology & Network Awareness: How to arrange transmission routes considering rack locality to avoid saturating inter-rack spine switches.
4. Stragglers & Failure Handling: Mechanisms for handling slow or failed nodes without blocking other workers.
5. API & Control Plane: Basic control plane architecture for coordination, peer discovery, and integrity checks.