Save a list of files over SFTP to a text file

Question

I have provided my own answer below.

We have a number of bash scripts that work fine using SFTP, what I would like to do is to simply re-direct an ls or dir of a folder to a file on our SFTP Server..

We can run this as a cron job or I can run it manually.. I can view the list of files from the remote server, but I want to generate the list of files from the remote server to the local server in a text file..

Here is what I have for the bash script so far.. The fourth echo listed below is the line in question.

#!/bin/bash

localpath=/home/localacct/somepath
remotelocation=/home/account/logs/archive
remotehost=' [email protected]'

    echo  > $localpath/list.sftp
    echo "cd $remotelocation " >> $localpath/list.sftp
    echo "dir *.* " >> $localpath/list.sftp
    echo "dir *.* > $localpath/dirlist.txt " >> $localpath/list.sftp
    echo "bye " >> $localpath/list.sftp
    sftp -b $localpath/list.sftp $remotehost

exit

Answer 1

The sftp command is very limited. If you can't make it do what you want, you can use another approach, which is to mount the remote directory with the SSHFS filesystem. SSHFS uses SFTP as the transport protocol, so the server side just sees an SFTP client. You need to be able to use FUSE on the client side.

With SSHFS, you mount the remote directory onto an existing, empty directory and use ordinary commands.

mkdir remote
sshfs "$remotehost:$remotelocation" remote
cd remote

echo *.* >"$localpath/dirlist.txt"

fusermount -u remote
rmdir remote

Answer 2

This works by far the best:

echo 'ls' | sftp hostname

You can forward the output into a file by

echo 'ls' | sftp hostname > /tmp/mylist.txt

Answer 3

The dir command within the sftp client does not support redirection. Example below, showing how it does nothing.

sftp> pwd
Remote working directory: /var/tmp/foodir
sftp> lcd /var/tmp/foodir
sftp> dir *.*
foo.txt
sftp> dir *.* >dirlist.txt
foo.txt
sftp> dir
foo.txt
sftp>

Man page for sftp confirms.

ls [-1afhlnrSt] [path]

Display a remote directory listing of either path or the current directory if path is not specified.  path may contain glob(3) characters and may match multiple files.

The following flags are recognized and alter the behaviour of ls accordingly:

-1

Produce single columnar output.
-a
List files beginning with a dot (‘.’).
-f
Do not sort the listing. The default sort order is lexicographical.
-h
When used with a long format option, use unit suffixes: Byte, Kilobyte, Megabyte, Gigabyte, Terabyte, Petabyte, and Exabyte in order to reduce the number of digits to four or fewer using powers of 2 for sizes (K=1024, M=1048576, etc.).
-l
Display additional details including permissions and ownership information.
-n
Produce a long listing with user and group information presented numerically.
-r
Reverse the sort order of the listing.
-S
Sort the listing by file size.
-t
Sort the listing by last modification time.

Answer 4

in case anyone finds rsync doesn't work (for reports.paypal.com in my case) I found the following script using expect works:

#!/usr/bin/expect
spawn sftp [email protected]:/ppreports/outgoing
expect "password:"
send "XXXXXXXXXX\n"
expect "sftp>"
log_file -noappend RemoteFileList.txt
send "ls -1\n"
expect "sftp>"
log_file
send "!sed -i '' '/ls -1/d' ./RemoteFileList.txt\n"
expect "sftp>"
send "!sed -i '' '/sftp>/d' ./RemoteFileList.txt\n"
expect "sftp>"
send "bye\n"
interact

This is on Mac. On other systems the sed lines might do without the first '' arguments.

Answer 5

You can also use lftp:

lftp -c 'connect sftp://user:password@host/dir; ls' > list.txt

Or

lftp -e 'ls;quit' sftp://user:password@host/dir > list.txt

lftp also has a find command for recursive listing.

Passing passwords on the command line is generally a bad idea. Alternative is to use ssh key authentication if possible (the password above will be ignored if key authentication is available, so you make it the empty string). Or you can pass it via the environment:

 LFTP_PASSWORD=password lftp --env-password -e 'ls;quit' \
   sftp://user:password@host/dir > file.list

Or pass the commands via stdin, here using a here document (implemented with deleted tempfiles or pipes depending on the shell):

lftp <<'EOF' > file.list
connect sftp://user:password@host/dir
ls
EOF

Answer 6

There is another way which I wound up using.. Try using rsync.

We find the following:

• If you specify no local destination then a listing of the specified files on the remote server is provided.

The BASH script then becomes:

#!/bin/bash
localpath=/home/local-acct/path
remotelocation=/home/account/logs/archive
remotehost=' [email protected]'
rsync -avz $remotehost:/$remotelocation > $localpath/dirlist.txt
exit

and works great!

Answer 7

So you can use this method too (with different example) :

#!/bin/bash +x

fct()
{
        a=`sftp user@host<< EOL
        ls -la ..
EOL`
        echo $a
        echo $a > tsftpres.000
}

fct
fct > tsftpres.001

fct_2()
{
        sftp user@host << EOL
        ls -la ..
EOL
}

fct_2 > tsftpres.002

Answer 8

I can't add a comment but i post a new answer comment about my last answer.

Yes i know but it just an example of the method but i write this so quickly normally i prefer writing echo "${a}",i use ls -la .. because i wanted to test my sshd_config (ChrootDirectory) but i rewrite this better :

#!/bin/bash +x

# with a varaiable or directly in file
# Replace "user", "host" and filename with your value
function list_sftp()
{
    list=`sftp user@host << EOL
    ls -la
EOL`
    echo "${list}" > tsftpres-000.txt
}

list_sftp

function other_lsftp()
{
    sftp user@host << EOL
    ls -la
EOL
}

other_lsftp > tsftpres-001.txt