Part 3 of our series exploring TCP examines the TCP “Fast Open” option and what to look for when troubleshooting
TCP Fast Open (TFO) is an optional mechanism within TCP that lets endpoints that have established a full TCP connection in the past eliminate a round-trip of the handshake and send data right away. This speeds things up for endpoints that are going to keep talking to each other in the future and is especially beneficial on high-latency networks where time-to-first-byte is critical.
Follow along with these sample captures
As always, we've gathered up the captures mentioned in this article into this collection over on CloudShark
You know you can make collections just like this with CloudShark! Learn more...
Fast Open is defined in RFC 7413 which explains:
TFO allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, and saves up to one full round-trip time (RTT) compared to the standard TCP, which requires a three-way handshake (3WHS) to complete before data can be exchanged.
How does it work?
TCP Fast Open works by allowing the server to set a special “TFO” cookie to use for your later connections. The next time you connect, you can include data in the SYN packet, and the server can include its response data in the first ACK. Normally, the client and server would need to do a full TCP handshake every time before sending application data. The RFC goes into detail on the protocol sequence in section 3.
Interestingly, TLS 1.3 has a very similar feature for saving a round trip, called “0-RTT” or zero round trip, which we’ll go into more detail about in our series on TLS.
What does it look like?
Here’s an initial SYN packet from this example capture.
Notice that the TCP Length (the
Len: field) is zero, since there’s no data
in the initial handshake:
If you expand the TCP Options you’ll see that the TCP Fast Open option is included (ALT or OPTION-Click to expand everything). Here the client sends the Fast Open Cookie Request option asking the server give it a TFO cookie for next time.
Here’s the resulting SYN+ACK from the server:
If you expand the TCP Options you’ll again see the TCP Fast Open option, this
time including the cookie value of
The TCP stream then continues as normal after that.
Second connection using the TFO cookie
So what happens when the same client makes a new connection to the same server using TCP Fast Open? Take a look at the first packet in this capture:
Even without expanding any fields, you immediately see two things:
- The TCP Length field (
Len:) is bigger than zero.
- There’s HTTP data in the packet!
The server recognizes the valid TFO cookie, accepts the payload data,
and replies with its SYN+ACK. In this case, the
Ack: field value is
set to the total length of the TCP payload (
76 in this case).
Why use TCP Fast Open?
The biggest reason to use TFO is to get that first chunk of data faster. Our captures were on our local network so the results aren’t quite as dramatic as they would be over say a mobile network, but still significant. If you look at the timestamps of the packets in the first and second captures, the first capture doesn’t get an HTTP response back from the server until 0.00447 seconds have passed. In the second capture, it only takes 0.00184 seconds. That’s more than twice (2.42 times) as fast!
Trouble with TCP Fast Open
What happens when the TFO cookie is invalid or not supported? Section 4.2.2 of RFC 7413 gets really deep into the details of implementation and handling of error cases. Here are some scenarios at the packet layer of what you might see.
If the server ends up rebooting or clears its TFO cookies for any reason, the client will still try to use a cookie that is now invalid. Take a look at the SYN from the client in this capture:
The client still tries to include data with the initial SYN, and uses the old cookie value. However, the SYN+ACK from the server is now different:
You can see that the value of
Ack: is 1 (not 76) indicating that the additional
payload wasn’t accepted. However, the server includes a new TFO cookie in the
options with a value of
Take a look at the next packet after the SYN+ACK in that capture:
Retransmitting the data
Section 4.2.2 of RFC 7413 states "The client SHOULD retransmit any unacknowledged data in the first ACK packet in step 2. The data exchange will start after the handshake like a regular TCP Connection."
Here we see that the client sent the same data in the initial SYN, but now again in the third packet of the normal 3-way handshake. CloudShark marks this as a retransmission as well because the data had already been sent in the first packet.
This might look strange, as in the normal flow of TCP, it’s usually:
You can clearly see the difference in this ladder diagram of the conversation.
Server stops supporting TFO
What happens if the server decides to stop accepting TFO? Take a look at the ladder diagram of this conversation.
This example is similar to the one above but the server does not respond with a new TFO cookie. In this case, since the client can still connect, it continues to try using the cookie. We aren’t getting any benefits from TFO, but this isn’t causing any problems either.
Can’t connect using TFO
What if a device along the path doesn’t understand the TFO option and starts blocking the client’s connection attempts? What if the client’s traffic is taking a new route to the server, or an older piece of equipment was pulled from the “back closet” and added to the network? Sending SYN packets with a TCP length > 0 is unusual, and could cause issues with older devices, including dropping the packet!
Take a look at the 11th frame of this capture:
Here the client tries to connect and includes a cookie, but this packet is never gets a response. One second later it is retransmitted:
Unlike before, the retransmission does not contain the TFO option or additional data, just in case that’s what’s causing the packet to get dropped before the server receives it.
How do I find TFO in my own traffic?
Have you seen TCP Fast Open traffic in your captures? Here are some display filters to help you find samples of this in your own files or when diagnosing connection problems:
- Filter for option:
tcp.option_kind == 34
- Filter for request:
- Filter for tcp server:
- Filter for tcp syn packet using a cookie:
tcp.flags.syn && tcp.payload && tcp.options.tfo.cookie
Remember in CloudShark you can also use the DeepSearch feature to scan your capture list for the presence of TFO. Captures that have packets matching these filters will be highlighted for you.
Did we say this was going to be a quick and easy post about TCP Fast Open? It turns out even this feature can get pretty complicated, but like window scaling, TFO demonstrates the flexibility of TCP to evolve to the networking world’s changing needs.
Do you have a scenario with TCP Fast Open that has looked strange? Feel free to send it our way or use CloudShark collections to help tell the whole story.