Socket: How does connect/accept use the 3-way handshake, if they ever do?

Question

I have a TCP server and multiple clients attempting to connect to that server almost simultaneously. I notice that:

1. On the client side, connect may return 0 even though the 3-way handshake has not been completed yet.

2. On the server side, accept may not return even after the 3-way handshake has been completed.

To illustrate both points, here are the Wireshark traces (server is listening on port 1234):

1. Here are the Wireshark traces for the case where client's connect returns 0 even though the 3-way handshake has not been completed (missing last SYN from client):

// calls ::connect ...
59507 → 1234 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
1234 → 59507 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM=1
// ... and ::connect returned 0, despite the above 2 lines 
// forming an incomplete handshake. Why?

// after the ::connect, client calls ::send to send 8 bytes ...
59507 → 1234 [PSH, ACK] Seq=1 Ack=1 Win=262656 Len=8

// ... but we got reset by peer
1234 → 59507 [RST] Seq=1 Win=0 Len=0

// ... and as expected, ::send returned 10054 (WSAECONNRESET)

2. Server's accept doesn't return even after the 3-way handshake has been completed:

// server calls ::accept ...
59643 → 1234 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
1234 → 59643 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 WS=256 SACK_PERM=1
59643 → 1234 [ACK] Seq=1 Ack=1 Win=262656 Len=0
// At this point, we have a complete handshake, but server's ::accept doesn't return. Why?

// In the next line, server sends a RST. This doesn't cause the server's ::accept call 
// to return an INVALID_SOCKET, which doesn't seem unreasonable (since nothing is accepted), 
// nor is this the point of the question, but I am including the RST trace here 
// for completeness. The main point is that it's as if server's ::accept is oblivious 
// to the successful handshake.
1234 → 59643 [RST] Seq=1 Win=0 Len=0

Questions:

1. When, exactly, does connect return? Apparently the traces above suggests that connect can return (and declare success with return value of 0) even if the handshake is incomplete.

2. When, exactly, does accept return? The traces above suggests that the completion of a 3-way handshake isn't good enough for accept to return, which is puzzling.

So, it's as if connect doesn't care about checking the handshake, while accept is so strict that a successful handshake is not good enough for it to return?

Answer 1

So to answer the question,

• When does connect() and accept() return?

I am quoting from the source "Unix Network Programming" -- By W. Richard Stevens. The connect() returns after the first two steps of the handshake. While the accept() returns when an entry is placed in the completed queue. The meaning of queues are given as:

When the client requests a TCP connection, the server's TCP stack creates an entry in the incomplete queue, then the 3-way TCP handshake is managed by the server TCP stack. The connection is moved from the incomplete queue to the completed queue when the last message of the TCP handshake arrives (ACK or PiggyBacked ACK).

For other questions raised in the comments:

• How does the backlog (2nd parameter of the listen()) affect the queue size?

backlog is not equal to the queue size, the queue size set by the backlog varies from system to system. Below table lists the backlog and corresponding queue values for some systems:

• What does the server do when it gets SYN, and the queue is full?

If the queues are full when a client SYN arrives, TCP ignores the arriving SYN (pp. 930-931 0f TCPv2); it does not send an RST. Because the condition is considered temporary, and the client may retry after sometime, hoping to find the place in the queue. If instead RST is sent, then it is ambiguous whether the server queues are full or the server is not listening on that port.