Table of Contents
How to disconnect hung ssh session in Linux. disconnect stuck ssh session in Unix, Terminate stuck ssh session. kill an unresponsive ssh session in Unix. Kill stuck ssh session. Terminate stuck ssh session in Linux. kill ssh session in Linux. Close ssh connection. Terminate an unresponsive ssh session in LInux. Automatically kill or disconnect hung ssh session using ServerAliveInterval. Automatically kill a PSSH session using timeout in Linux. Disconnect hung pssh session in linux after certain time period. Kill stuck ssh session automatically in Linux. Automatically disconnect hung ssh session in Unix.
In this article we will cover various examples around the topic to kill stuck ssh session i.e to disconnect hung ssh session in Linux, but if you wish to automatically disconnect an idle session then I have written another article wherein I have shared the steps and examples to automatically disconnect an idle ssh session in Linux.
What is an unresponsive SSH session?
By unresponsive SSH session we mean that a SSH connection is not responding or has become unresponsive between the host and the client. This situation can arise due to various reason, most likely due to network fluctuations between the server and the client. You can list all the active SSH connections and then check the idle time to get their current status. You can also get their PID from the remote host to manually kill the unresponsive ssh session.
Terminate stuck ssh session
There is a "secret" keyboard shortcut to force an unresponsive ssh session to exit.
From the frozen session terminal, hit these keys in order: Enter
→ ~
→ .
The tilde (only after a newline) is recognised as an escape sequence by the ssh client, and the period tells the client to terminate it's business without further ado.
From the man page of SSH
ESCAPE CHARACTERS
When a pseudo-terminal has been requested, ssh supports a number of functions through the use of an escape character.
A single tilde character can be sent as ~ or by following the tilde by a character other than those described below. The escape character must always follow a newline to be interpreted as special. The escape character can be changed in configuration files using the EscapeChar configuration directive or on the command line by the -e option.
The supported escapes (assuming the default ‘~’) are:
~. Disconnect.
~^Z Background ssh.
~# List forwarded connections.
..... output trimmed
For example:
Here I have created an active session from node1 to node2 using user 'deepak
'. Now this session is in hung state.
node1:~ # ssh -q deepak@node2 Password: Last login: Fri May 31 10:47:39 2019 from 10.43.22.85 [deepak@node2 ~]$
Here i have pressed "Enter
" followed by ~
and .
[deepak@node2 ~]$ <-- Here I pressed Enter [deepak@node2 ~]$ node1:~ # ^ | Here I pressed ~ (tilde) followed by . (dot)
As you see my hung session is terminated and I have returned back to my localhost.
Enter → ~ → .
will get you back to your local session, Enter → ~ → ~ → .
will leave you in Machine1, and Enter → ~ → ~ → ~ → .
will leave you in Machine2. This works for other escape sequences as well, such as moving the ssh session to background temporarily. The above works for any level of nesting, by just adding more tilde's.For example:
Here I have made an active SSH session from node1 to node2 and from node2 to node3 i.e. node1
→ node2
→ node3
node1:~ # ssh -q deepak@node2 Password: Last login: Fri May 31 11:04:17 2019 from node1 [deepak@node2 ~]$ ssh -q deepak@10.43.138.3 Password: Last login: Sun May 26 13:55:55 2019 from node2
Now here I will disconnect stuck SSH session from node3 to node2 instead of node1. So I will hit Enter
→ ~
→ ~
→ .
[deepak@node3 ~]$ <-- Here I pressed Enter [deepak@node3 ~]$ [deepak@node2 ~]$ <-- And I have been disconnected from node3 and my session is at node1 -> node2 ^ | Here I pressed ~ ~ (tilde) two times followed by . (dot)
Automatically disconnect hung SSH session
Now in the above example you were manually terminating the stuck ssh session, but to automatically disconnect hung SSH session or when you are automating a task and your SSH process gets stuck, in such cases it is better if the SSH itself disconnect hung SSH session itself rather than someone manually intervening and killing the process.
In this case you can use "ServerAliveInterval
" to automatically disconnect the hung SSH session.
From the man page:
ServerAliveCountMax Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive. The default value is 3. If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at the default, if the server becomes unresponsive, ssh will disconnect after approximately 45 seconds. ServerAliveInterval Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.
So using ServerAliveInterval
you can set a timeout value which will be used if the provided SSH session becomes unresponsive. You can use this either via command line or by creating a ssh_config
file inside the respective user's home directory.
Method 1:
While initiating a SSH connection use "-o ServerAliveInterval=XX
" along with the SSH command. here replace "XX
" with your suitable timeout value.
For example:
I am running a SSH session from node1 to node2, and will disconnect the NIC interface. Now this is not the best scenario to have a hung session but this will do the trick to explain our steps and example. Now here I have provided an interval of 5 seconds and TCP packet will be sent two times before disconnecting the session.
node1:~ # ssh -vvv -i /export/home/sufuser/.ssh/id_rsa -o ServerAliveInterval=5 -o ServerAliveCountMax=2 root@node2 "/sbin/ifdown bond0"
From the debug logs printed on the screen, there are two debug packets which are sent before disconnecting the session
debug3: send packet: type 80 debug3: send packet: type 80 debug3: send packet: type 1 packet_write_wait: Connection to node2 port 22: Broken pipe
Now here as expected my session my session was automatically disconnected after sending 2 TCP packets at an interval of 5 seconds.
Method 2:
You can configure these values per user by creating a new file ssh_config
inside user's home directory. For example if my user's home directory is /home/deepak/
then create following file
# touch /home/deepak/.ssh/ssh_config
Add below content
Host * ServerAliveCountMax 3 ServerAliveInterval 10
Here * is used to refer for any matching host. If you wish to restrict it for a certain host then provide the IP/hostname.
Method 3:
Lastly you can update /etc/ssh/ssh_config
and add the same values. This was the configuration will not be at user level instead it will be at system level
Add below content
# vim /etc/ssh/ssh_config Host * ServerAliveCountMax 3 ServerAliveInterval 10
Use timeout to kill SSH session
You can additionally use timeout command to disconnect hung SSH session. To define a timeout value for the active session.
Here the benefit is that if for automation you are using ssh command and you do not wish the automation script to get stuck during execution then you can use timeout command to kill the session after certain period of time.
From the man page of timeout:
-k, --kill-after=DURATION also send a KILL signal if COMMAND is still running this long after the initial signal was sent
For example:
I have created a script to trap SIGTERM and capture the progress of timeout
node1:~ # cat /tmp/script.sh #!/bin/bash trap 'echo "`date`: SIGTERM received!"' 15 echo "`date`: Starting" sleep 20 echo "`date`: Stage Two" sleep 20 echo "`date`: Finished"
Adding the -k 12
switch tells timeout to send a SIGKILL to the process 12 seconds after the initial SIGTERM. Again, the "SIGTERM received
" message is displayed after 3 seconds as a result of the SIGTERM signal. Twelve seconds after that, the whole script is killed. It does not complete the second 20-second
sleep, and it does not display the Finished
message after it. This is a more forceful way of dealing with the timeout.
node1:~ # timeout -k 12 3 /tmp/script.sh ; date
Fri May 31 13:26:45 IST 2019: Starting
Terminated
Fri May 31 13:26:48 IST 2019: SIGTERM received!
Fri May 31 13:26:48 IST 2019: Stage Two
Killed
Fri May 31 13:27:00 IST 2019
Similarly you can utilise timeout with SSH command. Assuming you know that your SSH command execution will take 10 seconds then you can define a timeout of 20 seconds as the hard period for killing the session
For example:
node1:~ # time timeout -k 25 20 ssh -q -i /export/home/sufuser/.ssh/id_rsa root@node2 "sleep 1m"
real 0m20.008s
user 0m0.012s
sys 0m0.008s
Disconnect PSSH session
PSSH is another tool to perform SSH (parallel) instead of sequential. Now in PSSH there is an internal argument which can be used to kill the active session after a certain time period. You need not be dependent on any additional tool to disconnect the SSH session.
From the man page of pssh:
-t timeout --timeout timeout Make connections time out after the given number of seconds. With a value of 0, pssh will not timeout any connections.
For example:
Execute pssh using below syntax to perform a timeout automatically after 30 seconds
node1:~ # pssh --timeout 30 -l root -H node2 sleep 40
[1] 14:21:48 [FAILURE] node2 Timed out, Killed by signal 9
So as expected the PSSH session was automatically killed with the timeout signal after the provided timeout value.
Lastly I hope the steps from the article to disconnect hung SSH session or unresponsive SSH session on Linux was helpful. So, let me know your suggestions and feedback using the comment section.
References:
why do consoles sometimes hang forever when ssh connection breaks
Superb!
the only article I found on google (not internet, but google 🙂 )
which correctly describes how to kill a very long SSH session by setting/defining hard limits.
I’ve been in this business for quite a while and have always been annoyed about not knowing how to kill only one nested ssh session and not the whole chain. I’m happy I ran across your page!
This is a great. I was always annoyed with hung ssh sessions and had to kill them using ps -a, kill and other tricks. This article explains the solution perfectly and I am saved from performing so many not so comfortable actions just in order to get rid of a hung ssh session.
This article is worthy of recognition and comment. I found this material attention-grabbing and engrossing. This is well-scripted and highly informative.