In this article I will share sample SFTP scripts to transfer files covering below scenarios in Linux or Unix environment
- Use batch file with SFTP shell script without prompting password
- Automate SFTP using shell script with password
- Use batch file with SFTP shell script with password example
- SFTP command example in unix shell script with password
- Use passwordless sftp in script
What is SFTP Batch File and How to automate SFTP using shell script with password in Batch Mode?
- Batch File in SFTP can be a plan text format file which contains a series of commands.
- These commands are read by SFTP in the sequential order from top to down
- This batch file is used to automate SFTP file transfers, can also be combined with scripts to transfer files without any prompts
- Use
-b
with sftp to provide the batch file name and path and to use batch mode for sftp file transfers in scripts - Batch mode reads a series of commands from an input batchfile instead of stdin.
- Since batch mode lacks interactions, you can use batch file with SFTP shell script without prompting password using sftp authorized_keys
- Batch file can also be used to automate SFTP using shell script with password but you may need additional tools such as sshpass, expect etc to avoid password prompts and user interaction
- A batchfile of ‘
-
’ may be used to indicate standard input. - sftp will abort if any of the following commands fail: get, put, reget, reput, rename, ln, rm, mkdir, chdir, ls, lchdir, chmod, chown, chgrp, lpwd, df, symlink, and lmkdir.
- Termination on error can be sup‐pressed on a command by command basis by prefixing the command with a ‘-’ character (for example,
-rm /tmp/blah*
).
So with the above explanation we know using batch file we can automate SFTP file transfers with scripts for both the situations
- SFTP shell script without prompting password i.e. passwordless SFTP (using sftp
authorized_keys
) - SFTP shell script with password (using
expect
,sshpass
or similar tools to pass the password)
SFTP shell script without prompting password (passwordless SFTP)
Step 1: Setup SFTP and Configure SFTP authorized_keys
I have already shared step by step guide to setup SFTP in my previous article with chroot jail and . So I will use the same server to create and use SFTP shell script without prompting password and performing passwordless SFTP.
In the same article I have also added a chapter to configure SFTP authrorized_keys to enable passwordless SFTP so I will not repeat the same steps here.
Step 2: Create SFTP script to transfer files without prompting password
I have taken the template of below script from Wicked Cool Scripts and modified it to transfer files without prompting password. You can follow the additional comments I have added in the script to understand the overall functionality.
[root@server1 ~]# cat /tmp/sftpsync.sh #!/bin/bash # # Description: # Get user, remote server, source directory with absolute path and # remote directory details as input to sync the latest added files # with remote server # # Use batch file with SFTP shell script without prompting password # using SFTP authorized_keys # ################################################################## # Create SFTP batch file tempfile="/tmp/sftpsync.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 if [ $# -eq 0 ] ; then echo "Usage: $0 user host path_to_src_dir remote_dir" >&2 exit 1 fi # Collect User Input user="$1" server="$2" remote_dir="$4" source_dir="$3" timestamp="$source_dir/.timestamp" # Without source and remote dir, the script cannot be executed if [[ -z $remote_dir ]] || [[ -z $source_dir ]]; then echo "Provide source and remote directory both" exit 1 fi echo "cd $remote_dir" >> $tempfile # timestamp file will not be available when executed for the very first time if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in $source_dir/* do if [ -f "$filename" ] ; then # Place the command to upload files in sftp batch file echo "put -P \"$filename\"" >> $tempfile # Increase the count value for every file found count=$(( $count + 1 )) fi done else # If timestamp file found then it means it is not the first execution so look out for newer files only # Check for newer files based on the timestamp for filename in $(find $source_dir -newer $timestamp -type f -print) do # If found newer files place the command to upload these files in batch file echo "put -P \"$filename\"" >> $tempfile # Increase the count based on the new files count=$(( $count + 1 )) done fi # If no new files found the do nothing if [ $count -eq 0 ] ; then echo "$0: No files require uploading to $server" >&2 exit 1 fi # Place the command to exit the sftp connection in batch file echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." # Main command to use batch file with SFTP shell script without prompting password sftp -b $tempfile "$user@$server" echo "Done. All files synchronized up with $server" # Create timestamp file once first set of files are uploaded touch $timestamp # Remove the sftp batch file rm -f $tempfile exit 0
Step 3: Verification
Now since we have our SFTP script to transfer files without password using sftp authorized_keys
, it is time to verify the script functionality:
My Lab Environment
SFTP Client:
- Hostname:
server1.example.com
- IP Address:
10.10.10.12
- Source directory which contains files to be transferred:
/src_dir
SFTP Server:
- Hostname:
server2.example.com
- IP Address:
10.10.10.13
- Remote directory which contains files to be collected:
dest_dir
- Chroot Jail path:
/opt/sftp-jails/deepak
We will create some new files under our source directory on server1
[root@server1 ~]# touch /src_dir/file{1..3}
Next use our sftp script to transfer files from server1
to server2
[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
Synchronizing: Found 3 files in local folder to upload.
sftp> cd dest_dir
sftp> put -P "/src_dir/file3"
sftp> put -P "/src_dir/file1"
sftp> put -P "/src_dir/file2"
sftp> quit
Done. All files synchronized up with server2
So all the files were transferred successfully using batch file with SFTP script without prompting password.
Next if we try to re-run the script, then you see since there are no newer files to transfer, the script does nothing.
[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
/tmp/sftpsync.sh: No files require uploading to server2
So we will create some more new files under our source directory on server1
# touch /src_dir/file{4..5}
Next re-run the sftp script to transfer files.
# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
Synchronizing: Found 2 files in local folder to upload.
sftp> cd dest_dir
sftp> put -P "/src_dir/file5"
sftp> put -P "/src_dir/file4"
sftp> quit
Done. All files synchronized up with server2
As expected the new files were automatically identified and transferred using batch file with SFTP script without prompting password.
Automate SFTP using shell script with password (Using Expect)
Now you can automate SFTP using shell script with password in combination with multiple third party tools such as expect
or sshpass
. In our example I will show SFTP command example in Unix shell script with password using expect
Step 1: Install Expect on client node
Assuming that you already have a SFTP server configured, the first step would be to install expect on your client node (which for us is server1
). By default expect
is not installed on all the Linux and Unix variant. Since I am using RHEL/CentOS 7/8 variant, I will install expect using yum/dnf
[root@server1 ~]# yum -y install expect
Step 2: SFTP command example in Unix shell script with password
Next we will create a script in combination with bash and expect to automate SFTP using shell script with password.
user_pwd="$5"
So the user has to pass the user's password as last input argument. The script can also be enhanced to pass an encrypted password and then decrypt it within the script to increase the security.
Below is a sample expect script to execute SFTP command example in Unix shell script with password:
expect -c "
spawn sftp -o "BatchMode=no" -b "$tempfile" "$user@$server"
expect -nocase \"*password:\" { send \"your_password\r\"; interact }
"
We can place this expect script in our existing /tmp/sftpsync.sh which we created earlier
[root@server1 ~]# cat /tmp/sftpsync.sh #!/bin/bash # # Description: # Get user, remote server, source directory with absolute path and # remote directory details as input to sync the latest added files # with remote server # # Execute SFTP command example in Unix shell script with password # Using batch file with expect ################################################################## tempfile="/tmp/sftpsync.$$" count=0 <Output trimmed> if [ $count -eq 0 ] ; then echo "$0: No files require uploading to $server" >&2 exit 1 fi echo "quit" >> $tempfile expect -c " spawn sftp -o "BatchMode=no" -b "$tempfile" "$user@$server" expect -nocase \"*password:\" { send \"My_Passw0rd\r\"; interact } " echo "Synchronizing: Found $count files in local folder to upload." touch $timestamp rm -f $tempfile exit 0
Step 3: Verification
Now since we have our SFTP command example in Unix shell script with password using expect and batch file, it is time to verify the script functionality:
My Lab Environment
SFTP Client:
- Hostname:
server1.example.com
- IP Address:
10.10.10.12
- Source directory which contains files to be transferred:
/src_dir
SFTP Server:
- Hostname:
server2.example.com
- IP Address:
10.10.10.13
- Remote directory which contains files to be collected:
dest_dir
- Chroot Jail path:
/opt/sftp-jails/deepak
We will create some new files under our source directory on server1
[root@server1 ~]# touch /src_dir/file{6..7}
Next execute the script to verify if automate SFTP using shell script with password is working:
[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir spawn sftp -o BatchMode=no -b /tmp/sftpsync.13391 deepak@server2 deepak@server2's password: sftp> cd dest_dir sftp> put -P "/src_dir/file6" sftp> put -P "/src_dir/file7" sftp> quit Synchronizing: Found 2 files in local folder to upload. Done. All files synchronized up with server2
As expected the new files are transferred to server2's
destination directory. So our SFTP command example in Unix shell script with password using expect and batch file is working.
If we re-run the script without adding new files:
[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
/tmp/sftpsync.sh: No files require uploading to server2
Since there are no new files on our source directory, SFTP command example in Unix shell script with password executed but no files were transferred.
Lastly I hope the steps from the article to automate SFTP using shell script with password on Linux and Unix was helpful. So, let me know your suggestions and feedback using the comment section.
Related Searches: SFTP command example in Unix shell script with password, SFTP shell script without prompting password, sftp script to transfer files, sftp script with password authentication, sftp batch file password example, automate sftp using shell script with password in Unix