PSSH is short abbreviation for Parallel Secure SHell or Parallel SSH. pssh is a program for executing ssh in parallel on a number of hosts. It provides features such as sending input to all of the processes, passing a password to ssh, saving output to files, and timing out.
The PSSH_NODENUM and PSSH_HOST environment variables are sent to the remote host. The PSSH_NODENUM variable is assigned a unique number for each ssh connection, starting with 0 and counting up. The PSSH_HOST variable is assigned the name of the host as specified in the hosts list.
PSSH is not part of RHEL or CentOS repo, so you will need to manually download and install the rpm from EPEL repo. For the sake of this demo I have created a password less configuration between multiple nodes on my setup.
1. Pass list of hosts using a file
You can create a file with list of hosts which can be used as an input for PSSH.
# cat /tmp/host_file.txt root@10.43.138.2:22 root@10.43.138.3:22 root@10.43.138.9:22
Now we will call PSSH for all the hosts from this file to execute date command.
# /bin/pssh -h /tmp/host_file.txt date [1] 19:29:01 [SUCCESS] root@10.43.138.3:22 [2] 19:29:01 [SUCCESS] root@10.43.138.2:22 [3] 19:29:01 [SUCCESS] root@10.43.138.9:22
So as you see the tool was successfully able to connect all the provided hosts and the EXIT STATUS for all the hosts was SUCCESS.
2. Pass list of hosts manually
If you only have few hosts list on which you wish to execute certain commands, then you can manually pass the info
# /bin/pssh -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date [1] 19:39:00 [SUCCESS] 10.43.138.3 [2] 19:39:00 [SUCCESS] 10.43.138.2 [3] 19:39:00 [SUCCESS] 10.43.138.9
3. Print inline output per host
You may have observed that for both the above examples, the EXIT STATUS was SUCCESS but yet the tool never printed any output on the screen. Use '-i' to display standard output and standard error as each host completes.
In the below example as you see we get an output for every host once the execution completes.
# /bin/pssh -i -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date [1] 19:39:00 [SUCCESS] 10.43.138.3 Sun Dec 16 19:39:00 IST 2018 [2] 19:39:00 [SUCCESS] 10.43.138.2 Sun Dec 16 19:39:00 IST 2018 [3] 19:39:00 [SUCCESS] 10.43.138.9 Sun Dec 16 19:39:00 IST 2018
4. Prompt for password
Since we have enabled password less communication between our hosts, the tool does not prompts for password. Using '-A' the tool will prompt for a password and pass it to ssh. The password may be used for either to unlock a key or for password authentication. The password is transferred in a fairly secure manner (e.g., it will not show up in argument lists). However, be aware that a root user on your system could potentially intercept the password.
# /bin/pssh -A -i -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date Warning: do not enter your password if anyone else has superuser privileges or access to your account. Password: [1] 20:14:06 [SUCCESS] 10.43.138.3 Sun Dec 16 20:14:06 IST 2018 [2] 20:14:06 [SUCCESS] 10.43.138.2 Sun Dec 16 20:14:06 IST 2018 [3] 20:14:06 [SUCCESS] 10.43.138.9 Sun Dec 16 20:14:06 IST 2018
5. Storing the STDOUT
Using '-o' or '--outdir' argument you can save standard output to files in the given directory. Filenames are of the form [user@]host[:port][.num] where the user and port are only included for hosts that explicitly specify them. The number is a counter that is incremented each time for hosts that are specified more than once.
# /bin/pssh -i -o /tmp/out/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date [1] 19:44:04 [SUCCESS] 10.43.138.3 Sun Dec 16 19:44:04 IST 2018 [2] 19:44:04 [SUCCESS] 10.43.138.2 Sun Dec 16 19:44:04 IST 2018 [3] 19:44:04 [SUCCESS] 10.43.138.9 Sun Dec 16 19:44:04 IST 2018
Next check /tmp/out to get the STDOUT result.
# ll /tmp/out/ total 12 -rw-r----- 1 root root 29 Dec 16 19:44 10.43.138.3 -rw-r----- 1 root root 29 Dec 16 19:44 10.43.138.2 -rw-r----- 1 root root 29 Dec 16 19:44 10.43.138.9
# cat /tmp/out/10.43.138.3 Sun Dec 16 19:44:04 IST 2018
6. Storing the STDERR
Using '-e' or '--errdir' you can save standard error to files in the given directory. Filenames are of the same form as with the '-o' option.
I will repeat the same command
# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root date [1] 19:46:46 [SUCCESS] 10.43.138.3 Sun Dec 16 19:46:46 IST 2018 [2] 19:46:46 [SUCCESS] 10.43.138.2 Sun Dec 16 19:46:46 IST 2018 [3] 19:46:46 [SUCCESS] 10.43.138.9 Sun Dec 16 19:46:46 IST 2018
But since the EXIT STATUS for all the commands were SUCCESS, the STDERR content is NULL.
# ll /tmp/err/ total 0 -rw-r----- 1 root root 0 Dec 16 19:46 10.43.138.2 -rw-r----- 1 root root 0 Dec 16 19:46 10.43.138.3 -rw-r----- 1 root root 0 Dec 16 19:46 10.43.138.9
So for the demo let me try to get a negative output. Here I am trying to run a command 'datet' which does not exists. So this means our PSSH will throw error
# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root -x '-q -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes' datet [1] 19:48:19 [FAILURE] 10.43.138.3 Exited with error code 127 Stderr: bash: datet: command not found [2] 19:48:19 [FAILURE] 10.43.138.2 Exited with error code 127 Stderr: bash: datet: command not found [3] 19:48:19 [FAILURE] 10.43.138.9 Exited with error code 127 Stderr: bash: datet: command not found
As expected we have received an EXIT CODE other than 0 and now if we check /tmp/err then we see the same is captured for every host.
# ls -l /tmp/err/ total 12 -rw-r----- 1 root root 31 Dec 16 19:48 10.43.138.3 -rw-r----- 1 root root 31 Dec 16 19:48 10.43.138.2 -rw-r----- 1 root root 31 Dec 16 19:48 10.43.138.9
While /tmp/out is empty since there was no STDOUT for this operation on any of the host
# ll /tmp/out/ total 0 -rw-r----- 1 root root 0 Dec 16 19:48 10.43.138.2 -rw-r----- 1 root root 0 Dec 16 19:48 10.43.138.3 -rw-r----- 1 root root 0 Dec 16 19:48 10.43.138.9
STDOUT and STDERR value at the same time per host.
7. Use SSHD options with PSSH
Now ideally PSSH accepts only certain list of options which is supports. But this does not means you cannot use SSHD arguments with PSSH.
With '-x' you can pass extra SSH command-line arguments (see the ssh(1) man page for more information about SSH arguments). This option may be specified multiple times. The arguments are processed to split on whitespace, protect text within quotes, and escape with backslashes.
To pass a single SSHD argument
# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root -x 'StrictHostKeyChecking=no' date [1] 19:46:46 [SUCCESS] 10.43.138.3 Sun Dec 16 19:46:46 IST 2018 [2] 19:46:46 [SUCCESS] 10.43.138.2 Sun Dec 16 19:46:46 IST 2018 [3] 19:46:46 [SUCCESS] 10.43.138.9 Sun Dec 16 19:46:46 IST 2018
To pass multiple SSHD argument
# /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root -x '-q -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o PubkeyAuthentication=yes' date [1] 19:53:56 [SUCCESS] 10.43.138.3 Sun Dec 16 19:53:56 IST 2018 [2] 19:53:56 [SUCCESS] 10.43.138.2 Sun Dec 16 19:53:56 IST 2018 [3] 19:53:56 [SUCCESS] 10.43.138.9 Sun Dec 16 19:53:56 IST 2018
8. Get list of hosts with SUCCESS and FAILED exit status
Now with PSSH one problem I faced was when we use this tool in scripting, we get a consolidated EXIT STATUS but we do not have host specific exits status by default. So if you were running a shell script with PSSH then you may not know out of 10 hosts which ones were successful and which one failed.
Now you can use -o and -e as an option to analyse this but this would require little scripting. I have written below piece of code in shell script which collects the lst of hosts and prints a SUMMARY with success and failed list of hosts.
function get_pssh_hosts {
err_dir=$1
out_dir=$2
tmp_f_hosts=$(mktemp /tmp/tmp_f_hosts.XXX)
tmp_s_hosts=$(mktemp /tmp/tmp_s_hosts.XXX)
find $err_dir -type f -not -empty -print | rev | cut -d / -f 1 | rev >> $tmp_f_hosts 2>&1
find $err_dir -type f -empty -print | rev | cut -d / -f 1 | rev >> $tmp_s_hosts 2>&1
if [ ! -z $out_dir ];then
for host in `cat $tmp_s_hosts`;do
[[ ! -f $out_dir/$host ]] && exit_with_error "Unable to get the exit status for $host"
# If file is empty then put to failed hosts
if [ ! -s $out_dir/$host ];then
if ! egrep -q ^$host$ $tmp_f_hosts; then
/bin/echo "$host" >> $tmp_f_hosts
sed -i "/^$host$/d" $tmp_s_hosts
fi
else
# If file is not empty then search for ERROR string, if found means the execution failed
if ! grep -q ERROR $out_dir/$host;then
if ! grep -q "^$host$" $tmp_s_hosts;then
/bin/echo "$host" >> $tmp_s_hosts
fi
# if no ERROR string found in the log then put in success and avoid duplicates
else
/bin/echo "$host" >> $tmp_f_hosts
sed -i "/^$host$/d" $tmp_s_hosts
fi
fi
done
fi
sed -i '/^$/d' $tmp_s_hosts
sed -i '/^$/d' $tmp_f_hosts
success_hosts=`cat $tmp_s_hosts | tr 'n' ' '`
failed_hosts=`cat $tmp_f_hosts | tr 'n' ' '`
rm -f $tmp_f_hosts
rm -f $tmp_s_hosts
}
function print_summary {
/bin/echo ""
/bin/echo "#####"
/bin/echo "#"
/bin/echo -e "# e[01;37mSUMMARY:e[0m"
/bin/echo "#"
/bin/echo -e "# e[01;32mSuccess:e[0m $success_hosts"
/bin/echo -e "# e[01;31mFailed:e[0m $failed_hosts"
/bin/echo "#"
/bin/echo "#####"
/bin/echo ""
}
ERROR for every command which fails with PSSH so I can use the regex to get success and failed list of hosts.How this works?
err_dir=$(mktemp -d /tmp/errdir.XXX) out_dir=$(mktemp -d /tmp/outdir.XXX) /bin/pssh -i -o /tmp/out/ -e /tmp/err/ -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root `datet || echo "ERROR"` get_pssh_hosts $err_dir $out_dir print_summary
Now here we get output like below in STDERR
# grep ERROR /tmp/err/10.43.138.3 bash: ERROR: command not found ##### # # SUMMARY: # # Success: # Failed: 10.43.138.2 10.43.138.3 10.43.138.9 # #####
So our script can easily detect and distinguish between success and failed scenario
9. Use PSCP to copy files from one server to multiple server
pscp is a program for copying files in parallel to a number of hosts. It provides features such as passing a password to scp, saving output to files, and timing out. This tool also supports almost all the options as with PSSH.
In the below example I am copying a file available on my server at '/tmp/temp.txt' on the client nodes under '/home/deepak/'
# /bin/pscp.pssh -H "10.43.138.2 10.43.138.3 10.43.138.9" -l root /tmp/temp.txt /home/deepak/ [1] 20:22:47 [SUCCESS] 10.43.138.3 [2] 20:22:47 [SUCCESS] 10.43.138.2 [3] 20:22:47 [SUCCESS] 10.43.138.9
10. Perform PSSH without password prompt
You can follow Perform SSH public key authentication with PSSH (without password) in Linux to get the steps to perform SSH to multiple hosts using private public key authentication. You can use this method in scripts to automate some tasks without getting any password prompt.
Summary
PSSH is a very handy tool when you want to perform any task on multiple target nodes in parallel. You can easily integrate this tool with any other wrapper script in any other programming language. I have already given one example of shell script where I am customizing the output and storing the list of success and failed hosts.
Further Reading
can some one help me how to pass sudo su – password or su – password ?
very helpful guide, thanks for your help