System Design: Pushing a Multi-Terabyte Snapshot to Thousands of Nodes

A freshly built search-index snapshot is 2 TB. It needs to land on every node in a serving fleet of somewhere between 200 and 2,000 machines before those nodes can start answering queries against it. Every machine — including the one that currently holds the snapshot — has a 25 Gbps network interface, and that capacity is shared between whatever the machine is sending and whatever it's receiving at the same time.

Assume no particular network topology: any two machines can talk to each other at full line rate, with no rack or switch bottleneck to worry about. Begin with the physics of the problem: pin down the fastest this could conceivably finish. Only after that's nailed down should you sketch a first, deliberately crude design and iterate it toward that physical limit.

Part 1 — The floor

Work out the minimum possible time to get all 2 TB onto every node, separately for a 200-node fleet and a 2,000-node fleet. Be explicit about:

• how you convert 2 TB and 25 Gbps into comparable units, and what that implies for a single machine receiving the whole snapshot alone;
• what changes if a machine's 25 Gbps cap applies to combined traffic in both directions at once, versus a looser reading where sending and receiving each get a full 25 Gbps independently;
• an argument based on the fleet's total send-and-receive capacity versus the total number of bits that must arrive somewhere, not just a single machine's link speed;
• why, once machines are allowed to forward data they've already received to each other, the bound you derive barely depends on whether the fleet has 200 or 2,000 machines.

Part 2 — The design

Describe a naive baseline (the source machine sends the full snapshot directly to every node) and work out how long it actually takes under that baseline for both fleet sizes — then show why it's far from the Part 1 floor, and redesign toward something that approaches it. Your design should address:

• splitting the snapshot into chunks and letting any node that has a chunk serve it to others, rather than everything flowing through the original source;
• how a node decides which chunks to fetch and from which peers, so that popular "rare" chunks don't become a bottleneck;
• what happens when a handful of nodes have unusually slow or flaky links (stragglers) and shouldn't hold up the rest of the fleet;
• what happens when a node fails mid-transfer, and how the system notices and recovers without losing data that only that node had;
• how the fleet knows a node is fully, correctly synced, and how the new snapshot is cut over into serving traffic only once enough of the fleet is ready.

Clarifying questions worth raising

• Is the 25 Gbps figure a combined ceiling across sending and receiving together, or is it available separately to each direction?
• Is the snapshot served from one machine, or from an object store that can itself serve many machines in parallel?
• Does every node need the complete 2 TB, or only a shard of it?
• Must the fleet keep serving the old snapshot while the new one is still arriving?