Table of Contents
Last week, a friend sent me a network trace (pcap) file and asked me to check if there was something wrong with the TLS communication between the server and client. After spending sometime analyzing the packets with Wireshark, I figured out packet fragmentation was the culprit behind the troubled communication. I will review the packet capture below, but before that we need to talk about Maximum Transmission Unit (MTU) first.
What are MTU and MSS?
MTU can be defined as the maximum length of a data packet that is transmitted on a network or medium. Data is typically transmitted in packet format and therefore it is essential to determine the packet size to ensure packet transmission efficiency. When large size packets are used:
- Each packet contains more data and the communication efficiency gets higher.
- The delay in packet transmission increases.
- The probability of bit errors gets higher and in case of packet loss, the retransmissions get higher.
When small size packets are used:
- Each packet contains less data.
- It reduces network latency.
- The number of packets increase in the network and the endpoints need to process more packets
- It consumes more bandwidth.
To improve network forward efficiency, the appropriate MTU size must be found. The default MTU size varies according to the link medium type. Default MTU sizes for some mediums are below.
Maximum Segment Size (MSS) refers to the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. It can be simply defined as maximum TCP payload. During the TCP 3-way handshaking, the peers let the other side know the size of MSS it accepts. Following screenshot shows that the client informs the server about its MSS in the option field.
The figure below shows the default MTU size for Ethernet.
A packet gets fragmented when the packet size exceeds the MTU on any point in the network path. The sender can specify if any network device in between communication peers is allowed to fragment the packet or not with “Don't fragment” bit in IP header. When the bit is set to zero (0), it means the packet can be fragmented as it exceeds the MTU of the link, but when the bit is set to one (1), the packet can not be fragmented when it exceeds the MTU of the link and will be dropped. Some devices that fragment the packet may inform the sender about fragmentation with an ICMP “Fragmentation needed” packet. Most of security devices ignore sending the ICMP packet. When this happens, it becomes extremely difficult to identify the problem. For better understanding, I will prepare following network topology and show the fragmentation in details with Wireshark.
Step-1: The Trivial File Transfer Protocol (TFTP) sends a packet size of 544 bytes without setting “Don’t fragment” bit to 1, which means that any network device can fragment the packet on any point in the network path.
Step-2: When the packet arrives Router0, it checks if the packet size exceeds its MTU size, which is set to 400 bytes. Since the packet size is greater than 400 bytes and “Don’t fragment” bit is not set to 0, the router will fragment it. After breaking the packet into two smaller packets, it needs to preserve the original “Identification” field for both of the new packet so that when the client receives the packets, it can combine them properly.
Step-3: When the client receives the first packet, it checks the “More fragments” bit. As seen below, the flag is set to 1, which means the packet was fragmented. The receiver collects all packets with 0x0000 Identification header until it sees “More fragments” bit set to zero. Then it combines all of them.
Step-4: After receiving the last packet, the client defragments all the packets. Following figure shows that the last packet has the same identification header with “More fragments” bit set to zero.
Troubleshoot Packet Fragmentation with Wireshark
At first glance in our pcap, we can see there is a troubled communication between the client and server. After 6 retransmissions, the server gives up and finishes the conversation in packet number 19. If you are not familiar with how TLS or MTLS communication works then I would first recommend you to read Analyze TLS and mTLS Authentication with Wireshark
Step-1: After a successful TCP 3-way handshaking, the client requests for “START TLS” and server accepts that, then the client sends a “Client Hello” packet to start TLS handshaking. As seen below, the client desires to use the previous session. Because of that the complete TLS handshaking was not made and previous session was used.
Step-2: In the packet number 11, we can see the server have already sent 140 bytes and it is also sending new data which is size of 2134 bytes. When the client receives that, it should acknowledge (ACK) for 140+2134=2274. Let’s see if the client will acknowledge it in the next packet.
Step-3: With packet number 12, the client updates its window but it also acknowledges the server for the amount of the data it received, which is 140 bytes. This does not sound OK, because the server already sent 2134 bytes in the previous packet and it seems to be the packet has been lost, which indicates that there is something wrong along the path.
Step-4: After the server noticing the packet did not reach the client, it lowers the TCP payload and sends a payload of 1560 bytes. There is still no answer from the client and the server keeps sending the same packet (the retransmissions) again and again for 6 seconds.
Step-5: The server gives up and finishes the connection in the packet number of 19.
Step-6: It seems that the packet does not reach to the client. The packet must be dropped in the network somehow. This is not an ordinary packet loss and we need to find the reason so we will focus more on the packet. I will expand the IP header like below. As it can be seen, the “Don't fragment” bit is 1, which means this packet can not be fragmented in the network. In the other hand, the total length is 2174 bytes, which exceeds the default Ethernet MTU size. This is a great clue for resolving the problem. The path MTU must be checked. Indeed, after changing the network MTU to appropriate size, the problem disappeared.
Path MTU Discovery
Path MTU Discovery is a method through which a host can detect a path MTU smaller than its interface MTU. There are some tools on the internet that allow you to discover it, but we will use the simple “ping” command. We will craft a packet with “Don't fragment” bit set and send it to a destination. When a network device notices the crafted packet size is larger that its MTU, it will drop the packet and send back an ICMP “Fragmentation Needed” packet. The packet will also contain the MTU that path supports.
We will craft a packet size of 1470 bytes and set “Don't fragment” bit with “ping” command like below. The packet will be forwarded if the next hop MTU is greater than the packet size and the packet will be discarded if the next hop MTU is smaller than the packet size and an ICMP “Fragmentation needed” will be sent back, which will contain the next hop MTU size. The command takes the flags below.
-M do: Sets “Don't fragment” bit.
-c: Number of the packets will be crafted.
-s: Size of the packet(s).
As seen in the screenshot above, the path MTU is 1492. I also started Wireshark during sending the crafted packet. Below shows the crafted packet.
After receiving the packet, my router figures out that its MTU is smaller than the packet size and the “Don't fragment” bit is set to 1. Because of that, it drops the packet and sends back an ICMP “Fragmentation needed” packet, which contains the routers MTU. Following figure shows the packet details.
The same result can be achieved in Windows as well. The exact command can not be used, because of Windows uses different flags for the command. The command is below.
ping 188.8.131.52 -f -l 1470
- -f: It sets “Don’t Fragment” bit.
- -l: It sets the size of the packet.
Sometimes, Maximum Transmit Unit (MTU) in the network can be lower than size of a transmitted packet and network devices (routers, firewalls, switches etc.) may need to fragment the packet. When the “Don't fragment” bit (flag) is set to 1 in the IP header, the network devices can not fragment the packet and have to drop the packet. Some devices may inform the sender about dropping the packet with an ICMP “Fragmentation Needed” packet and some not, which are mostly security devices.
Related Keywords: fragmented ip protocol wireshark udp 17, observe ip fragmentation using tcpdump and wireshark, how to tell if ip datagram is fragmented, wireshark fragment offset, wireshark fragmented ip protocol filter, ip fragmentation questions, how to check packet fragmentation in wireshark, wireshark packet examples, wireshark packet analysis example, how to find fragmented packets in wireshark, how to check fragmentation in wireshark, fragmentation packets in wireshark