Troubleshooting TLS Failures using Wireshark


Written by - Nurten Dogan
Reviewed by - Deepak Prasad

In one of the previous articles Analyze TLS and mTLS Authentication with Wireshark, we explored how SSL/TLS handshake works and analyzed SSL/TLS record types in Wireshark. TLS handshaking employs three subprotocols that are used to allow peers to agree upon security parameters for the record layer, to authenticate themselves, to instantiate negotiated security parameters, and to report error conditions to each other. These protocols are:

  1. The Handshake Protocol, which performs the initial key negotiation.
  2. The ChangeCipherSpec Protocol, which changes the encryption system currently in use.
  3. The Alert Protocol, which sends important messages about the state of the SSL/TLS connection from one side to the other.

In this article, we will focus on the Alert Protocol to troubleshoot TLS issues.

 

Alert Protocol

This protocol uses alert messages to notify the peers about the status of the TLS connection. The messages fall into two levels (categories): Warning and Fatal.

Each alert message consists of two parts: An Alert Level and an Alert Description. Following table shows a brief description for the levels.

Alert Level Level Name Alert Description (Meaning)
1 Warning It is a warning, which does not cause a connection termination.
2 Fatal Requires immediate termination of the connection.

Alert messages convey the severity of the message (warning or fatal) and a description of the alert.  Alert messages with a level of fatal result in the immediate termination of the connection. Like other messages, alert messages are encrypted and compressed, as specified by the current connection state. For troubleshooting, we will analyze messages in two groups: Closure and Error Alerts.

 

Closure Alerts

Every TLS connection ends with this alert message, which notifies the recipient that the sender will not send any more messages on this connection.  Either party may initiate a close by sending this alert. Any data received after a closure alert is ignored. Unless some other fatal alert has been transmitted, each party is required to send this alert (close notify) before closing the write side of the connection. The screenshot below shows a typical data transfer over a TLS connection.

Troubleshooting TLS Failures using Wireshark

 

The data transfer follows the steps below:

  1. The TCP connection is established with the TCP 3-way handshaking.
  2. For securing the data, the TLS session is created with the TLS handshake.
  3. The data gets transferred securely.
  4. No data to send and the TLS session is terminated with Closure Alert.
  5. The final step is closing the TCP connection.

Since this message is encrypted, we can not see the real alert message. So how do we actually know this is a closure alert? Without decrypting the message, it is not possible, but we can check the pattern above (5 steps) see if it is match. There is another way, which is decrypting the traffic. Here are some articles on decrypting TLS (https://www.golinuxcloud.com/wireshark-decrypt-ssl-tls-tutorial/  https://www.golinuxcloud.com/decrypt-rdp-traffic-wireshark-frida/ )

After decryption, we will see the output below.

Troubleshooting TLS Failures using Wireshark

 

This alert level is warning, which is completely normal to be seen during a connection termination. While troubleshooting, you should not focus on Closure Alerts.

 

Error Alerts

Error handling in the TLS Handshake protocol is very simple.  When an error is detected, the detecting party sends a message to the other party.  Upon transmission or receipt of a fatal alert message, both parties immediately close the connection.

If an alert with a level of warning is sent and received, generally the connection can continue normally.  If the receiving party decides not to proceed with the connection, it sends a fatal alert to terminate the connection.  Given this, the sending party cannot, in general, know how the receiving party will behave.  Therefore, warning alerts are not very useful when the sending party wants to continue the connection, and thus are sometimes omitted.  For example, if a peer decides to accept an expired certificate (perhaps after confirming this with the user) and wants to continue the connection, it would not generally send a Certificate Expired alert.

RFC 5246 defines following error alerts:

 

Alert Message Alert Code Alert Description
close notify 0 This notifies the recipient that the sender will not send any more
messages on this connection. Either party may initiate a close by sending this alert.
unexpected message 10 An inappropriate message was received.  This alert is always fatal and should never be observed in communication between proper implementations.
bad record mac 20 This alert is returned if a record is received with an incorrect MAC. This message is always fatal and should never be observed in communication between proper implementations.
decryption failed RESERVED 21 This alert was used in some earlier versions of TLS, and may have
permitted certain attacks against the CBC mode.
record overflow 22 A TLSCiphertext record was received that had a length more than 2^14+2048 bytes, or a record decrypted to a TLSCompressed record with more than 2^14+1024 bytes.  This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network)
decompression failure 30 The decompression function received improper input (e.g., data
that would expand to excessive length).  This message is always
fatal and should never be observed in communication between proper implementations.
handshake failure 40 Reception of a handshake_failure alert message indicates that the sender was unable to negotiate an acceptable set of security
parameters given the options available.  This is a fatal error.
no certificate RESERVED 41 This alert was used in SSLv3 but not any version of TLS.
bad certificate 42 A certificate was corrupt, contained signatures that did not
verify correctly, etc.
unsupported certificate 43  A certificate was of an unsupported type.
certificate revoked 44 A certificate was revoked by its signer.
certificate expired 45 A certificate has expired or is not currently valid.
certificate unknown 46 Some other (unspecified) issue arose in processing the
certificate, rendering it unacceptable.
illegal parameter 47 A field in the handshake was out of range or inconsistent with
other fields.  This message is always fatal.
unknown ca 48  A valid certificate chain or partial chain was received, but the
certificate was not accepted because the CA certificate could not be located or couldn't be matched with a known, trusted CA.  This message is always fatal.
access denied 49  A valid certificate was received, but when access control was
applied, the sender decided not to proceed with negotiation.  This message is always fatal.
decode error 50 A message could not be decoded because some field was out of the specified range or the length of the message was incorrect.  This message is always fatal and should never be observed in
communication between proper implementations (except when messages were corrupted in the network).
decrypt error 51  A handshake cryptographic operation failed, including being unable to correctly verify a signature or validate a Finished message. This message is always fatal.
export restriction RESERVED 60 This alert was used in some earlier versions of TLS.
protocol version 70 The protocol version the client has attempted to negotiate is
recognized but not supported.  (For example, old protocol versions might be avoided for security reasons.)  This message is always fatal
insufficient security 71 Returned instead of handshake_failure when a negotiation has
failed specifically because the server requires ciphers more
secure than those supported by the client.  This message is always fatal.
internal error 80 An internal error unrelated to the peer or the correctness of the
protocol (such as a memory allocation failure) makes it impossible to continue.  This message is always fatal.
user canceled 90  This handshake is being canceled for some reason unrelated to a
protocol failure.  If the user cancels an operation after the
handshake is complete, just closing the connection by sending a
close_notify is more appropriate.  This alert should be followed
by a close_notify.  This message is generally a warning.
no renegotiation 100 Sent by the client in response to a hello request or by the server
in response to a client hello after initial handshaking.  Either
of these would normally lead to renegotiation; when that is not
appropriate, the recipient should respond with this alert.  At
that point, the original requester can decide whether to proceed
with the connection.  One case where this would be appropriate is where a server has spawned a process to satisfy a request; the
process might receive security parameters (key length,
authentication, etc.) at startup, and it might be difficult to
communicate changes to these parameters after that point.  This message is always a warning.
unsupported extension 110 Sent by clients that receive an extended server hello containing
an extension that they did not put in the corresponding client
hello.  This message is always fatal.

 

Common TLS Handshake Failures

TLS handshake failure, which may be due to many reasons, is a common error. Sometimes troubleshooting them can be a frustrating. Thanks to Wireshark with decrypting TLS feature and great display filters, we can find them easily. Some common failures are below.

 

Certificate Expired

A certificate expires once its validity period is over. When one side receives the expired certificate, it sends this alert message. As seen below, the client notifies the server with a fatal error and closes the connection after receiving an expired certificate.

Troubleshooting TLS Failures using Wireshark

Sometimes, this error happens due to mismatched time synchronization.

 

Bad Certificate

This alert message is sent when there is a certificate mismatch between the client and server. I tried to connect to “wrong.host.badssl.com”, specifying the domain name in the TLS Client Hello packet with Server Name Indication field. However, the server sent me a certificate for “badssl.com” domain.  Due to this reason the client sent Bad Certificate alert message and terminated the connection.

Troubleshooting TLS Failures using Wireshark

 

Unknown CA

This error happens when a valid certificate chain or partial chain is received, but the certificate is not accepted because the CA certificate can not be located or matched with a known, trusted CA.  This message is always fatal, which terminates the TLS session.

Troubleshooting TLS Failures using Wireshark

 

Certificate Revoked

Revoking a certificate is a process of invalidating an issued SSL certificate. A revoked certificate is no longer trusted and can no longer be used for certificate-based encryption. A certificate is revoked mostly when its private key shows signs of being compromised. Following screenshot shows that the certificate has been revoked.

Troubleshooting TLS Failures using Wireshark

 

Handshake Failure

Reception of a handshake failure alert message indicates that the sender was unable to negotiate an acceptable set of security parameters given the options available. This is a fatal error. As seen in the screenshot below, the client sends a bunch of cipher suits, none of which is supported by the server.

Troubleshooting TLS Failures using Wireshark

Since the server supports only RC4-MD5 security parameters, it responses back with a handshake failure alert.

Troubleshooting TLS Failures using Wireshark

 

Protocol Version

The protocol version that one side has attempted to negotiate is recognized but not supported.  (For example, old protocol versions might be avoided for security reasons.)  This message is always fatal.

The following screenshot shows that the client negotiates for TLS version 1.2 whereas the server supports only version 1.0. Therefore, the client terminates the session with sending protocol version alert.

Troubleshooting TLS Failures using Wireshark

 

Filtering TLS Handshake Failures

There are times when we need to take a quick look at the TLS alerts for troubleshooting. Wireshark provides a display filter for this purpose. After applying “tls.record.content_type == 21” display filter, all alerts will be displayed in Wireshark like below.

Troubleshooting TLS Failures using Wireshark

 

Final Thoughts

With Alert Protocol, it is very easy to troubleshoot TLS handshake problems. However, some of the alerts are encrypted and we can not see the alerts details like level and message. The TLS traffic must be decrypted so that the alerts can be seen in clear text.

 

References

https://www.rfc-editor.org/rfc/rfc5246#section-7.2
https://www.oreilly.com/library/view/web-security-privacy/0596000456/apbs03.html
https://support.huaweicloud.com/intl/en-us/ccm_faq/ccm_01_0085.html

 

Views: 67

Nurten Dogan

He is proficient in System Administration, Python, Computer Network, Network Engineering, PHP, Web Testing, Penetration Testing, Wireshark, RADIUS, Cisco Router, TCP/IP, Kali Linux, OSPF, NPS, and Multiprotocol BGP. You can connect with him on LinkedIn.

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

Leave a Comment