Automate SFTP using shell script with password in Linux/Unix

In this article I will share sample SFTP scripts to transfer files covering below scenarios in Linux or Unix environment

  • Use batch file with SFTP shell script without prompting password
  • Automate SFTP using shell script with password
  • Use batch file with SFTP shell script with password example
  • SFTP command example in unix shell script with password
  • Use passwordless sftp in script

 

What is SFTP Batch File and How to automate SFTP using shell script with password in Batch Mode?

  • Batch File in SFTP can be a plan text format file which contains a series of commands.
  • These commands are read by SFTP in the sequential order from top to down
  • This batch file is used to automate SFTP file transfers, can also be combined with scripts to transfer files without any prompts
  • Use -b with sftp to provide the batch file name and path and to use batch mode for sftp file transfers in scripts
  • Batch mode reads a series of commands from an input batchfile instead of stdin.
  • Since batch mode lacks interactions, you can use batch file with SFTP shell script without prompting password using sftp authorized_keys
  • Batch file can also be used to automate SFTP using shell script with password but you may need additional tools such as sshpass, expect etc to avoid password prompts and user interaction
  • A batchfile of ‘-’ may be used to indicate standard input.
  • sftp will abort if any of the following commands fail: get, put, reget, reput, rename, ln, rm, mkdir, chdir, ls, lchdir, chmod, chown, chgrp, lpwd, df, symlink, and lmkdir.
  • Termination on error can be sup‐pressed on a command by command basis by prefixing the command with a ‘-’ character (for example, -rm /tmp/blah*).

 

So with the above explanation we know using batch file we can automate SFTP file transfers with scripts for both the situations

  1. SFTP shell script without prompting password i.e. passwordless SFTP (using sftp authorized_keys)
  2. SFTP shell script with password (using expect, sshpass or similar tools to pass the password)

 

SFTP shell script without prompting password (passwordless SFTP)

Step 1: Setup SFTP and Configure SFTP authorized_keys

I have already shared step by step guide to setup SFTP in my previous article with chroot jail and . So I will use the same server to create and use SFTP shell script without prompting password and performing passwordless SFTP.

In the same article I have also added a chapter to configure SFTP authrorized_keys to enable passwordless SFTP so I will not repeat the same steps here.

 

Step 2: Create SFTP script to transfer files without prompting password

I have taken the template of below script from Wicked Cool Scripts and modified it to transfer files without prompting password. You can follow the additional comments I have added in the script to understand the overall functionality.

[root@server1 ~]# cat /tmp/sftpsync.sh
#!/bin/bash
#
# Description:
# Get user, remote server, source directory with absolute path and 
# remote directory details as input to sync the latest added files
# with remote server
#
# Use batch file with SFTP shell script without prompting password
# using SFTP authorized_keys
#
##################################################################
# Create SFTP batch file
tempfile="/tmp/sftpsync.$$"  
count=0

trap "/bin/rm -f $tempfile" 0 1 15      

if [ $# -eq 0 ] ; then
  echo "Usage: $0 user host path_to_src_dir remote_dir" >&2
  exit 1
fi

# Collect User Input
user="$1"
server="$2"
remote_dir="$4"
source_dir="$3"

timestamp="$source_dir/.timestamp"

# Without source and remote dir, the script cannot be executed
if [[ -z $remote_dir ]] || [[ -z $source_dir ]]; then
   echo "Provide source and remote directory both"
   exit 1
fi

echo "cd $remote_dir" >> $tempfile
# timestamp file will not be available when executed for the very first time
if [ ! -f $timestamp ] ; then   
  # no timestamp file, upload all files
  for filename in $source_dir/*
  do
    if [ -f "$filename" ] ; then
      # Place the command to upload files in sftp batch file
      echo "put -P \"$filename\"" >> $tempfile
      # Increase the count value for every file found
      count=$(( $count + 1 ))   
    fi
  done
else   
  # If timestamp file found then it means it is not the first execution so look out for newer files only
  # Check for newer files based on the timestamp
  for filename in $(find $source_dir -newer $timestamp -type f -print)   
  do
    # If found newer files place the command to upload these files in batch file
    echo "put -P \"$filename\"" >> $tempfile  
    # Increase the count based on the new files
    count=$(( $count + 1 ))   
  done
fi
# If no new files found the do nothing
if [ $count -eq 0 ] ; then   
  echo "$0: No files require uploading to $server" >&2
  exit 1
fi
# Place the command to exit the sftp connection in batch file
echo "quit" >> $tempfile   

echo "Synchronizing: Found $count files in local folder to upload."
# Main command to use batch file with SFTP shell script without prompting password
sftp -b $tempfile "$user@$server"  
echo "Done. All files synchronized up with $server"
# Create timestamp file once first set of files are uploaded
touch $timestamp  

# Remove the sftp batch file
rm -f $tempfile  
exit 0

 

Step 3: Verification

Now since we have our SFTP script to transfer files without password using sftp authorized_keys, it is time to verify the script functionality:

My Lab Environment

SFTP Client:

  • Hostname: server1.example.com
  • IP Address: 10.10.10.12
  • Source directory which contains files to be transferred: /src_dir

SFTP Server:

  • Hostname: server2.example.com
  • IP Address: 10.10.10.13
  • Remote directory which contains files to be collected: dest_dir
  • Chroot Jail path: /opt/sftp-jails/deepak

 

We will create some new files under our source directory on server1

[root@server1 ~]# touch /src_dir/file{1..3}

Next use our sftp script to transfer files from server1 to server2

[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
Synchronizing: Found 3 files in local folder to upload.
sftp> cd dest_dir
sftp> put -P "/src_dir/file3"
sftp> put -P "/src_dir/file1"
sftp> put -P "/src_dir/file2"
sftp> quit
Done. All files synchronized up with server2

So all the files were transferred successfully using batch file with SFTP script without prompting password.

Next if we try to re-run the script, then you see since there are no newer files to transfer, the script does nothing.

[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
/tmp/sftpsync.sh: No files require uploading to server2

So we will create some more new files under our source directory on server1

# touch /src_dir/file{4..5}

Next re-run the sftp script to transfer files.

# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
Synchronizing: Found 2 files in local folder to upload.
sftp> cd dest_dir
sftp> put -P "/src_dir/file5"
sftp> put -P "/src_dir/file4"
sftp> quit
Done. All files synchronized up with server2

As expected the new files were automatically identified and transferred using batch file with SFTP script without prompting password.

 

Automate SFTP using shell script with password (Using Expect)

Now you can automate SFTP using shell script with password in combination with multiple third party tools such as expect or sshpass. In our example I will show SFTP command example in Unix shell script with password using expect

 

Step 1: Install Expect on client node

Assuming that you already have a SFTP server configured, the first step would be to install expect on your client node (which for us is server1). By default expect is not installed on all the Linux and Unix variant. Since I am using RHEL/CentOS 7/8 variant, I will install expect using yum/dnf

[root@server1 ~]# yum -y install expect

 

Step 2: SFTP command example in Unix shell script with password

Next we will create a script in combination with bash and expect to automate SFTP using shell script with password.

 

Below is a sample expect script to execute SFTP command example in Unix shell script with password:

expect -c "
spawn sftp -o "BatchMode=no" -b "$tempfile" "$user@$server"
expect -nocase \"*password:\" { send \"your_password\r\"; interact }
"

We can place this expect script in our existing /tmp/sftpsync.sh which we created earlier

[root@server1 ~]# cat /tmp/sftpsync.sh
#!/bin/bash
#
# Description:
# Get user, remote server, source directory with absolute path and 
# remote directory details as input to sync the latest added files
# with remote server
#
# Execute SFTP command example in Unix shell script with password
# Using batch file with expect
##################################################################

tempfile="/tmp/sftpsync.$$"
count=0

<Output trimmed>

if [ $count -eq 0 ] ; then
  echo "$0: No files require uploading to $server" >&2
  exit 1
fi

echo "quit" >> $tempfile

expect -c "
spawn sftp -o "BatchMode=no" -b "$tempfile" "$user@$server"
expect -nocase \"*password:\" { send \"My_Passw0rd\r\"; interact }
"
echo "Synchronizing: Found $count files in local folder to upload."

touch $timestamp
rm -f $tempfile
exit 0

 

Step 3: Verification

Now since we have our SFTP command example in Unix shell script with password using expect and batch file, it is time to verify the script functionality:

My Lab Environment

SFTP Client:

  • Hostname: server1.example.com
  • IP Address: 10.10.10.12
  • Source directory which contains files to be transferred: /src_dir

SFTP Server:

  • Hostname: server2.example.com
  • IP Address: 10.10.10.13
  • Remote directory which contains files to be collected: dest_dir
  • Chroot Jail path: /opt/sftp-jails/deepak

We will create some new files under our source directory on server1

[root@server1 ~]# touch /src_dir/file{6..7}

Next execute the script to verify if automate SFTP using shell script with password is working:

[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
spawn sftp -o BatchMode=no -b /tmp/sftpsync.13391 deepak@server2
deepak@server2's password:
sftp> cd dest_dir
sftp> put -P "/src_dir/file6"
sftp> put -P "/src_dir/file7"
sftp> quit
Synchronizing: Found 2 files in local folder to upload.
Done. All files synchronized up with server2

As expected the new files are transferred to server2's destination directory. So our SFTP command example in Unix shell script with password using expect and batch file is working.

If we re-run the script without adding new files:

[root@server1 ~]# /tmp/sftpsync.sh deepak server2 /src_dir dest_dir
/tmp/sftpsync.sh: No files require uploading to server2

Since there are no new files on our source directory, SFTP command example in Unix shell script with password executed but no files were transferred.

 

Lastly I hope the steps from the article to automate SFTP using shell script with password on Linux and Unix was helpful. So, let me know your suggestions and feedback using the comment section.

 

Related Searches: SFTP command example in Unix shell script with password, SFTP shell script without prompting password, sftp script to transfer files, sftp script with password authentication, sftp batch file password example, automate sftp using shell script with password in Unix

Leave a Comment

Please use shortcodes <pre class=comments>your code</pre> for syntax highlighting when adding code.