I have an expect script that connects to an SFTP site and uploads some files. The ID and password are contained in local text files. The expect script reads those files in as a variable and then passes it to the SFTP server.

The problem I am experiencing is we are being provided passwords with complexity that includes special characters ($\,!) etc. For example if the password is pass$word the $ passes as a special rather than a literal. Being this is being passed through a variable (multiple sites and IDs)

I am unable to use '$pw' as it will literally pass $pw to the server and using "$pw" sends special characters. I need it to pass the password exactly as is.

I have been escaping them out by hand so far (e.g. pass\$word) but that is tedious and I would like to do it script-o-matically.

The script looks like this (names and places changed to protect the innocent)

#!/bin/bash

home=/data/ftp/vf/drop/
un=`cat /home/vf/workflow/.udf/.u/.u$1`
pw=`cat /home/vf/workflow/.udf/.p/.p$1`
sl=`cd /home/vf/workflow/scheduler; grep $1 upload*|cut -d \: -f1`

/usr/bin/expect -c "
spawn /usr/bin/sftp -o KexDHMin=1024 $un@$sl.mysteriouslocation.com
set timeout -1
expect *Authentication
expect Password*
send \"$pw\r\"
expect sftp*
send \"mput /$home/$1/transfer/*\r\"
expect sftp*
send \"ls \r\"
expect sftp*
send \"bye\r\"
expect eof
exit

How can I either pass the password to the expect script so that it sends the literal characters and not give them special meaning? I don't think I can script a "fix" for the password file itself, meaning go in and every time it sees a special character, escape it out via sed or other means as the escape character itself is special and could end up in a loop.

Your help is greatly appreciated!

  • In a sh/bash script, whenever you're having to escape lots of quotes, think about here documents. – glenn jackman Dec 5 '17 at 18:27
  • instead of scripting the password w/ expect, could you instead use passwordless ssh keys for authentication? – ivanivan Dec 5 '17 at 20:53
  • Why not get expect to read the files? Tcl is a full programming language. – icarus Dec 5 '17 at 22:16
  • Thank you all for the help! Ivanivan, I would love to use keys but it is against the providers of the ftp server. – Keith Dec 6 '17 at 18:49
  • I'm trying to get my head around here documents, this is actually the first time I've heard of them. We use an external file to store the password, is it basically adding in the EOF at the end and the reverse redirection? – Keith Dec 6 '17 at 18:50
up vote 1 down vote accepted

A small Perl snippet can be used to escape (backslash) all ASCII characters not matching /[A-Za-z_0-9]/ by using the quotemeta function. The Perl command line is easily incorporated into your shell script:

#/bin/sh
pstr='$x%y!zpass' # plain, un-escaped string
estr=$(perl -e 'print quotemeta shift(@ARGV)' "${pstr}")
echo ${estr}      # show escaped string

produces:

\$x\%y\!zpass
  • This looks like a very elegant solution, thanks! If I could ask, being the expect script is separate from the script calling it, could this just as easily work inside the expect script? – Keith Dec 6 '17 at 18:55
  • In fact after testing this, this solution works perfectly for what I'm trying to do, thanks! – Keith Dec 6 '17 at 20:19

Pass the values as environment variables. That will also avoid the command injection vulnerabilities and showing the password in ps output!

#! /bin/sh -

home=/data/ftp/vf/drop
un=$(cat /home/vf/workflow/.udf/.u/.u"$1")
pw=$(cat /home/vf/workflow/.udf/.p/.p"$1")
sl=$(cd /home/vf/workflow/scheduler &&
  grep -Fe "$1" upload*|cut -d : -f1)

export home un sl pw

user=$1 /usr/bin/expect -c '
spawn /usr/bin/sftp -o KexDHMin=1024 $env(un)@$env(sl).mysteriouslocation.com
set timeout -1
expect *Authentication
expect Password*
send $env(pw)\r
expect sftp*
send "mput /$env(home)/$env(user)/transfer/*\r"
expect sftp*
send ls\r
expect sftp*
send bye\r
expect eof
exit'

(well, it could still be a command injection vulnerability if the variable contain text that would make sftp run command like !reboot).

  • Thank you for this answer! This is all pvlan'd and a dedicated pipe so we're not horribly worries about command line passwords per se. I tried implementing this solution however the expect script seems to only have a 70% success rate or so, so I'll have to troubleshoot it a little to figure out why. – Keith Dec 6 '17 at 18:52
  • @Keith, note that expect is a TCL interpreter, TCL being in general a lot more capable than shells as a language, so you should be able to do the whole thing in expect as well. – Stéphane Chazelas Dec 6 '17 at 19:39

You could put the expect program in a separate script (let's call it sftp.expect) and pass the username/password as arguments to it.

This does have the issue that command line arguments are visible to other users on the system, so this isn't the best way to deal with passwords, and you really don't want to do this to passwords on a shared system.

#!/usr/bin/expect -f
set un [lindex $argv 0];
set pw [lindex $argv 1];
set sl [lindex $argv 2];

spawn /usr/bin/sftp -o KexDHMin=1024 $un@$sl.mysteriouslocation.com
set timeout -1
expect *Authentication
expect Password*
send "$pw\r"

Then run that from the shell script with

#!/bin/bash
un=$(cat /home/vf/workflow/.udf/.u/.u"$1")
pw=...
sl=...
expect -f sftp.expect "$un" "$pw" "$sl"

Or if you want to keep the expect script within the same shell script, stick it in a here-document:

#!/bin/bash    
un=foo
# ...
expect -f - "$un" <<'EOF'
set un [lindex $argv 0];
send "user is: $un\r\n"
# ...
EOF

(Note the quotes around EOF, we don't want the shell to interpret the variables in the heredoc.)

  • great advice. +1 – glenn jackman Dec 5 '17 at 18:26
  • Oh, rats, passwords on the command line is bad (My excuse is that I use this on a system where that doesn't matter, so I totally forgot about the issue.) – ilkkachu Dec 5 '17 at 18:32
  • The expect script is a separate script that is being called from another script doing some file manipulation/processing etc. Would the here document be a separate document outside the expect script? – Keith Dec 6 '17 at 18:53

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.