How to kill or disconnect hung ssh session in Linux

How to disconnect hung ssh session in Linux. disconnect stuck ssh session in Unix, Terminate stuck ssh session. kill an unresponsive ssh session in Unix. Kill stuck ssh session. Terminate stuck ssh session in Linux. kill ssh session in Linux. Close ssh connection. Terminate an unresponsive ssh session in LInux. Automatically kill or disconnect hung ssh session using ServerAliveInterval. Automatically kill a PSSH session using timeout in Linux. Disconnect hung pssh session in linux after certain time period. Kill stuck ssh session automatically in Linux. Automatically disconnect hung ssh session in Unix.

In this article we will cover various examples around the topic to kill stuck ssh session i.e to disconnect hung ssh session in Linux, but if you wish to automatically disconnect an idle session then I have written another article wherein I have shared the steps and examples to automatically disconnect an idle ssh session in Linux.

What is an unresponsive SSH session?

By unresponsive SSH session we mean that a SSH connection is not responding or has become unresponsive between the host and the client. This situation can arise due to various reason, most likely due to network fluctuations between the server and the client. You can list all the active SSH connections and then check the idle time to get their current status. You can also get their PID from the remote host to manually kill the unresponsive ssh session.

Terminate stuck ssh session

There is a "secret" keyboard shortcut to force an unresponsive ssh session to exit.
From the frozen session terminal, hit these keys in order: Enter → ~ → . The tilde (only after a newline) is recognised as an escape sequence by the ssh client, and the period tells the client to terminate it's business without further ado.

From the man page of SSH

ESCAPE CHARACTERS
     When a pseudo-terminal has been requested, ssh supports a number of functions through the use of an escape character.

     A single tilde character can be sent as ~ or by following the tilde by a character other than those described below.  The escape character must always follow a newline to be interpreted as special.  The escape character can be changed in configuration files using the EscapeChar configuration directive or on the command line by the -e option.

     The supported escapes (assuming the default ‘~’) are:
     ~.      Disconnect.	 
     ~^Z     Background ssh.
     ~#      List forwarded connections.
	..... output trimmed

For example:

Here I have created an active session from node1 to node2 using user 'deepak'. Now this session is in hung state.

node1:~ # ssh -q  deepak@node2
Password:
Last login: Fri May 31 10:47:39 2019 from 10.43.22.85
[deepak@node2 ~]$

Here i have pressed "Enter" followed by ~ and .

[deepak@node2 ~]$    <-- Here I pressed Enter
[deepak@node2 ~]$ node1:~ #
			 ^
			 |
Here I pressed ~ (tilde) followed by . (dot)

As you see my hung session is terminated and I have returned back to my localhost.

For example:
Here I have made an active SSH session from node1 to node2 and from node2 to node3 i.e. node1 → node2 → node3

node1:~ # ssh -q  deepak@node2
Password:
Last login: Fri May 31 11:04:17 2019 from node1

[deepak@node2 ~]$ ssh -q deepak@10.43.138.3
Password:
Last login: Sun May 26 13:55:55 2019 from node2

Now here I will disconnect stuck SSH session from node3 to node2 instead of node1. So I will hit Enter → ~ → ~ → .

[deepak@node3 ~]$    <-- Here I pressed Enter
[deepak@node3 ~]$ [deepak@node2 ~]$ <-- And I have been disconnected from node3 and my session is at node1 -> node2
				^
				|
Here I pressed ~ ~ (tilde) two times followed by . (dot)

Automatically disconnect hung SSH session

Now in the above example you were manually terminating the stuck ssh session, but to automatically disconnect hung SSH session or when you are automating a task and your SSH process gets stuck, in such cases it is better if the SSH itself disconnect hung SSH session itself rather than someone manually intervening and killing the process.

In this case you can use "ServerAliveInterval" to automatically disconnect the hung SSH session.

From the man page:

     ServerAliveCountMax
             Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the server.  If this threshold
             is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session.  It is important to note that
             the use of server alive messages is very different from TCPKeepAlive (below).  The server alive messages are sent through the encrypted channel and
             therefore will not be spoofable.  The TCP keepalive option enabled by TCPKeepAlive is spoofable.  The server alive mechanism is valuable when the
             client or server depend on knowing when a connection has become inactive.

             The default value is 3.  If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at the default, if the server
             becomes unresponsive, ssh will disconnect after approximately 45 seconds.

     ServerAliveInterval
             Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel
             to request a response from the server.  The default is 0, indicating that these messages will not be sent to the server.

So using ServerAliveInterval you can set a timeout value which will be used if the provided SSH session becomes unresponsive. You can use this either via command line or by creating a ssh_config file inside the respective user's home directory.

Method 1:

While initiating a SSH connection use "-o ServerAliveInterval=XX" along with the SSH command. here replace "XX" with your suitable timeout value.

For example:
I am running a SSH session from node1 to node2, and will disconnect the NIC interface. Now this is not the best scenario to have a hung session but this will do the trick to explain our steps and example. Now here I have provided an interval of 5 seconds and TCP packet will be sent two times before disconnecting the session.

node1:~ # ssh -vvv -i /export/home/sufuser/.ssh/id_rsa -o ServerAliveInterval=5 -o ServerAliveCountMax=2 root@node2 "/sbin/ifdown bond0"

From the debug logs printed on the screen, there are two debug packets which are sent before disconnecting the session

debug3: send packet: type 80
debug3: send packet: type 80
debug3: send packet: type 1
packet_write_wait: Connection to node2 port 22: Broken pipe

Now here as expected my session my session was automatically disconnected after sending 2 TCP packets at an interval of 5 seconds.

Method 2:

You can configure these values per user by creating a new file ssh_config inside user's home directory. For example if my user's home directory is /home/deepak/ then create following file

# touch /home/deepak/.ssh/ssh_config

Add below content

Host *
 ServerAliveCountMax 3
 ServerAliveInterval 10

Here * is used to refer for any matching host. If you wish to restrict it for a certain host then provide the IP/hostname.

Method 3:

Lastly you can update /etc/ssh/ssh_config and add the same values. This was the configuration will not be at user level instead it will be at system level

Add below content

# vim /etc/ssh/ssh_config
Host *
 ServerAliveCountMax 3
 ServerAliveInterval 10

Use timeout to kill SSH session

You can additionally use timeout command to disconnect hung SSH session. To define a timeout value for the active session.

Here the benefit is that if for automation you are using ssh command and you do not wish the automation script to get stuck during execution then you can use timeout command to kill the session after certain period of time.

From the man page of timeout:

       -k, --kill-after=DURATION

              also send a KILL signal if COMMAND is still running
              this long after the initial signal was sent

For example:
I have created a script to trap SIGTERM and capture the progress of timeout

node1:~ # cat /tmp/script.sh
#!/bin/bash

trap 'echo "`date`: SIGTERM received!"' 15
echo "`date`: Starting"
sleep 20
echo "`date`: Stage Two"
sleep 20
echo "`date`: Finished"

Adding the -k 12 switch tells timeout to send a SIGKILL to the process 12 seconds after the initial SIGTERM. Again, the "SIGTERM received" message is displayed after 3 seconds as a result of the SIGTERM signal. Twelve seconds after that, the whole script is killed. It does not complete the second 20-second sleep, and it does not display the Finished message after it. This is a more forceful way of dealing with the timeout.

node1:~ # timeout -k 12 3 /tmp/script.sh ; date
Fri May 31 13:26:45 IST 2019: Starting
Terminated
Fri May 31 13:26:48 IST 2019: SIGTERM received!
Fri May 31 13:26:48 IST 2019: Stage Two
Killed
Fri May 31 13:27:00 IST 2019

Similarly you can utilise timeout with SSH command. Assuming you know that your SSH command execution will take 10 seconds then you can define a timeout of 20 seconds as the hard period for killing the session

For example:

node1:~ # time timeout -k 25 20  ssh -q -i /export/home/sufuser/.ssh/id_rsa root@node2 "sleep 1m"

real    0m20.008s
user    0m0.012s
sys     0m0.008s

Disconnect PSSH session

PSSH is another tool to perform SSH (parallel) instead of sequential. Now in PSSH there is an internal argument which can be used to kill the active session after a certain time period. You need not be dependent on any additional tool to disconnect the SSH session.

From the man page of pssh:

       -t timeout
       --timeout timeout
              Make connections time out after the given number of seconds.  With a value of 0, pssh will not timeout any connections.

For example:
Execute pssh using below syntax to perform a timeout automatically after 30 seconds

node1:~ # pssh  --timeout 30 -l root -H node2 sleep 40
[1] 14:21:48 [FAILURE] node2 Timed out, Killed by signal 9

So as expected the PSSH session was automatically killed with the timeout signal after the provided timeout value.

Lastly I hope the steps from the article to disconnect hung SSH session or unresponsive SSH session on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

References:
why do consoles sometimes hang forever when ssh connection breaks