Execute long running commands over SSH [SOLVED]

In the world of system administration, developers, and IT professionals, SSH (Secure Shell) has become an indispensable tool for managing remote systems securely and efficiently. However, when it comes to executing long running commands over SSH, it's crucial to adopt appropriate techniques to prevent interruptions, disconnections, and loss of progress. This article will provide an in-depth guide on how to execute long running commands over SSH effectively, ensuring your work proceeds smoothly and without any unforeseen hiccups.

We will explore various methods, best practices, and third-party tools to help you maintain stability and reliability while executing long running commands over SSH. By understanding how to handle and monitor these tasks, you can avoid common pitfalls and ensure your remote management tasks are completed successfully, even in the face of unstable connections or unexpected interruptions. With a focus on practical solutions and clear guidance, this article aims to empower you to tackle long running commands confidently, maximizing your productivity and effectiveness in the management of remote systems.

 

Different methods to execute long running commands over SSH

1. Using nohup (no hang-up): The nohup command allows a process to continue running in the background even after the SSH session is disconnected. It also prevents the process from receiving the SIGHUP signal when the terminal is closed.

   ssh user@host 'nohup command > output.log 2>&1 &'

2. Using screen: GNU Screen is a terminal multiplexer that allows you to run multiple terminal sessions in a single window, detach from them, and reattach later, even from a different terminal.

   ssh user@host'screen -S session_name -d -m command'

   To reattach the session: screen -r session_name

3. Using tmux: Tmux is another terminal multiplexer, similar to GNU Screen, that allows you to run multiple terminal sessions, detach from them, and reattach later. It is often considered more powerful and user-friendly than Screen.

   ssh user@host 'tmux new-session -d -s session_name "command"'
   

   To reattach the session:

   tmux attach-session -t session_name

4. Using disown: The disown command is a shell builtin that removes a job from the shell's job table, allowing it to continue running in the background even after the SSH session is disconnected.

   ssh user@host'command & disown'
   

5. Using setsid: The setsid command creates a new session and process group, allowing the specified command to run independently of the current session and process group.

   ssh user@host'setsid command > output.log 2>&1 < /dev/null &'

6. Combining ssh with byobu: Byobu is a text-based window manager and terminal multiplexer that is built on top of Screen or Tmux. It provides an enhanced interface and additional features for managing terminal sessions.

   ssh user@host'byobu new-session -d -s session_name; byobu send-keys -t session_name "command" Enter'

   To reattach the session:

   byobu attach-session -t session_name

 

1. Using nohup (no hang-up)

The nohup command allows a process to continue running in the background even after the SSH session is disconnected. It prevents the process from receiving the SIGHUP signal when the terminal is closed.

Syntax:

ssh user@host 'nohup command > output.log 2>&1 &'

This example will run the sleep command for 3600 seconds (1 hour) in the background, redirecting both stdout and stderr to output.log.

ssh user@host 'nohup sleep 3600 > output.log 2>&1 &'

This example will change to the /var/log directory and create a compressed archive (logs.tar.gz) containing all .log files, running in the background with output redirected to output.log.

ssh user@host 'nohup bash -c "cd /var/log; tar czf logs.tar.gz *.log" > output.log 2>&1 &'

 

2. Using screen

Description: GNU Screen is a terminal multiplexer that allows you to run multiple terminal sessions in a single window, detach from them, and reattach later, even from a different terminal.

Syntax:

ssh user@host 'screen -S session_name -d -m command'

This example will create a new detached Screen session called sleep_session and run the sleep command for 3600 seconds (1 hour) within it.

ssh user@host 'screen -S sleep_session -d -m sleep 3600'

This example will create a new detached Screen session called backup_session, change to the /var/log directory, and create a compressed archive (logs.tar.gz) containing all .log files within it. Additionally we will redirect the STDOUT and STDERR to /tmp/output.log on the server node.

ssh user@host 'screen -S backup_session -d -m sh -c "cd /var/log && tar czf /path/to/logs.tar.gz *.log > /path/to/output.log 2>&1"'

For Example:

ssh localhost 'screen -S backup_session -d -m sh -c "cd /var/log && tar czf /tmp/logs.tar.gz *.log > /tmp/output.log 2>&1"'

This command created the following logs inside /tmp/output.log

$ cat /tmp/output.log 
tar: boot.log: Cannot open: Permission denied
tar (child): /tmp/logs.tar.gz: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/logs.tar.gz: Wrote only 4096 of 10240 bytes
tar: Child returned status 2
tar: Error is not recoverable: exiting now

So let me try to use sudo to execute the same command over SSH:

ssh user@host 'screen -S backup_session -d -m sh -c "cd /var/log && tar czf /path/to/logs.tar.gz *.log > /path/to/output.log 2>&1"'

To list the screen session on remote server,

$ ssh user@host "screen -ls"
There is a screen on:
	4899.backup_session	(01/05/23 12:30:34 PM IST)	(Detached)
1 Socket in /run/screen/S-deepak.

To reattach the session:

ssh user@host 'screen -r session_name'

 

3. Using tmux

Description: Tmux is another terminal multiplexer, similar to GNU Screen, that allows you to run multiple terminal sessions, detach from them, and reattach later. It is often considered more powerful and user-friendly than Screen.

Syntax:

ssh user@host 'tmux new-session -d -s session_name "command"'

This example will create a new detached Tmux session called sleep_session and run the sleep command for 3600 seconds (1 hour) within it.

ssh user@host 'tmux new-session -d -s sleep_session "sleep 3600"'

This will create a new detached tmux session named backup_session and run the tar command inside it. The command's output, including any error messages, will be redirected to /tmp/output.log.

ssh user@host 'tmux new-session -d -s backup_session "cd /var/log && tar czf logs.tar.gz *.log > /tmp/output.log 2>&1"'

To verify if the tmux sessions are running, use the following command:

ssh user@host 'tmux ls'

To reattach to a specific session, use:

ssh user@host 'tmux attach-session -t backup_session'

To check the contents of the log file, use:

ssh user@host 'cat /tmp/output.log'

 

4. Using disown

Description: The disown command is a shell builtin that removes a job from the shell's job table, allowing it to continue running in the background even after the SSH session is disconnected.

Syntax:

ssh user@host 'command & disown'

This example will run the sleep command for 3600 seconds (1 hour) in the background and disown the process, allowing it to continue running after the SSH session is disconnected.

ssh user@host 'sleep 3600 & disown'

This example will change to the /var/log directory, create a compressed archive (logs.tar.gz) containing all .log files, and disown the process to allow it to continue running in the background after the SSH session is disconnected. The command's output, including any error messages, will be redirected to /tmp/output.log.

ssh user@host 'bash -c "cd /var/log; tar czf logs.tar.gz *.log > /tmp/output.log 2>&1 & disown"'

 

5. Using setsid

The setsid command creates a new session and process group, allowing the specified command to run independently of the current session and process group.

Syntax:

ssh user@host 'setsid command > output.log 2>&1 < /dev/null &'

This example will run the sleep command for 3600 seconds (1 hour) in a new session, redirecting both stdout and stderr to output.log.

ssh user@host 'setsid sleep 3600 > output.log 2>&1 < /dev/null &'

This example will change to the /var/log directory, create a compressed archive (logs.tar.gz) containing all .log files, and run the command in a new session with output redirected to output.log.

ssh user@host 'setsid bash -c "cd /var/log; tar czf logs.tar.gz *.log > /tmp/output.log 2>&1"'

 

6. Combining ssh with byobu

Description: Byobu is a text-based window manager and terminal multiplexer that is built on top of Screen or Tmux. It provides an enhanced interface and additional features for managing terminal sessions.

Syntax:

ssh user@host 'byobu new-session -d -s session_name; byobu send-keys -t session_name "command" Enter'

This example will create a new detached Byobu session called sleep_session and run the sleep command for 3600 seconds (1 hour) within it. The command's output, including any error messages, will be redirected to /tmp/output.log.

ssh user@host 'byobu new-session -d -s sleep_session; byobu send-keys -t sleep_session "sleep 3600 > /tmp/sleep_output.log 2>&1" Enter'

This example will create a new detached Byobu session called backup_session, change to the /var/log directory, and create a compressed archive (logs.tar.gz) containing all .log files within it. The command's output, including any error messages, will be redirected to /tmp/output.log.

ssh user@host 'byobu new-session -d -s backup_session; byobu send-keys -t backup_session "cd /var/log; tar czf logs.tar.gz *.log > /tmp/output.log 2>&1" Enter'

List all active byobu sessions:

ssh user@host 'byobu list-sessions'

Example

$ ssh localhost 'byobu list-sessions'
deepak@localhost's password: 
backup_session: 1 windows (created Mon May 1 12:50:44 2023)
sleep_session: 1 windows (created Mon May 1 12:49:11 2023)

If the session you want to clear (in this case, backup_session) is listed, you can kill the session using the following command:

$ ssh localhost 'byobu kill-session -t backup_session'
deepak@localhost's password: 

$ ssh localhost 'byobu list-sessions'
deepak@localhost's password: 
sleep_session: 1 windows (created Mon May 1 12:49:11 2023)

To reattach the session:

ssh user@host 'byobu attach-session -t session_name'

 

Summary

In summary, executing long-running commands over SSH is a crucial skill for system administrators, developers, and IT professionals who manage remote systems. Long-running commands are those that take a significant amount of time to complete and may continue running even if the SSH session is disconnected. By understanding and implementing the appropriate techniques, you can ensure the smooth and uninterrupted execution of these commands.

Methods such as nohup, setsid, byobu, and disown allow you to run commands in the background and detach them from the SSH session. These methods prevent the commands from receiving the SIGHUP signal when the session is closed and allow them to continue running independently. Additionally, using tools like screen or tmux provides terminal multiplexing functionality, enabling you to manage multiple terminal sessions and reattach to them as needed.

Redirecting the command's output to log files, helps capture any errors, progress, or other relevant information generated during the command execution. This allows for later analysis and troubleshooting.

 

References

Doing a long-running command on ssh