Katkat
Back

Explain the TCP Three-Way Handshake and Its Guarantees

ByteDanceSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

Tests in-depth understanding of the TCP three-way handshake, state transitions, sequence/acknowledgment arithmetic, and failure modes such as dropped final ACKs.

Prompt

In distributed systems and backend infrastructure, reliable transport protocols underpin almost all RPC and client-server communication. Explain the mechanism, state transitions, and edge cases of the Transmission Control Protocol (TCP) three-way handshake.

Your explanation should thoroughly address the following aspects:

  1. Step-by-Step Handshake & Sequence Arithmetic

    • Detail each message sent between Client and Server (SYN, SYN-ACK, ACK).
    • Trace the exact transition of socket states on both endpoints (e.g., LISTEN, SYN_SENT, SYN_RCVD, ESTABLISHED).
    • Explain how Initial Sequence Numbers (ISNs) are selected, why they are not simply initialized to 0, and how the SEQ and ACK fields are updated across each step.

  2. Handling the Dropped Final ACK

    • What happens if the third packet (the client's ACK) is lost in transit?
    • How do the client and server states differ immediately after the drop?
    • How is this scenario resolved if the client immediately starts sending payload data versus if the connection remains idle?

  3. Guarantees and Limitations

    • What does completing the three-way handshake guarantee to both parties?
    • What does it not guarantee (e.g., in terms of subsequent application-layer liveness, intermediate middlebox state, or half-open connections)?

Hints

Hint 1

Recall that TCP control flags like SYN consume one logical sequence number, which determines the value expected in the ACK field.

Hint 2

Consider the server's perspective when its SYN-ACK receives no reply: what timer triggers, and what happens if client data arrives while the server is still in SYN_RCVD?

Hint 3

Think about middleboxes, NAT timeouts, and silent client crashes: establishing a TCP session does not ensure ongoing end-to-end reachability without traffic or keepalives.

Follow-up questions

  • How does a SYN flood attack exploit the handshake state, and how do SYN cookies mitigate it without allocating server memory?
  • Why are three steps necessary and sufficient to establish mutual connection state, rather than two or four?
  • How does TCP Fast Open (TFO) modify the traditional three-way handshake to reduce connection latency for repeated connections?