3

I have provided my own answer below.

We have a number of bash scripts that work fine using SFTP, what I would like to do is to simply re-direct an ls or dir of a folder to a file on our SFTP Server..

We can run this as a cron job or I can run it manually.. I can view the list of files from the remote server, but I want to generate the list of files from the remote server to the local server in a text file..

Here is what I have for the bash script so far.. The fourth echo listed below is the line in question.

#!/bin/bash

localpath=/home/localacct/somepath
remotelocation=/home/account/logs/archive
remotehost=' account@1.1.1.1'

    echo  > $localpath/list.sftp
    echo "cd $remotelocation " >> $localpath/list.sftp
    echo "dir *.* " >> $localpath/list.sftp
    echo "dir *.* > $localpath/dirlist.txt " >> $localpath/list.sftp
    echo "bye " >> $localpath/list.sftp
    sftp -b $localpath/list.sftp $remotehost

exit
  • So what is this script not doing right? – 111--- Jul 9 '15 at 20:45
  • It is not writing the list (ls) of files to the text file. There is something wrong with the following: echo "dir . > $localpath/dirlist.txt " – Leptonator Jul 9 '15 at 20:46
  • Just curious and probably stating the obvious: you must use SFTP, right? This will not work for your use case? ssh $remotehost ls > $localpath/list.sftp – berto Jul 10 '15 at 1:19
  • you should write you answer as separate answer and approve it. – Archemar Jul 13 '15 at 14:04
  • I undeleted my previously posted Answer and is now visible. Thanks for the guidance. – Leptonator Jul 13 '15 at 14:14
0

I can't add a comment but i post a new answer comment about my last answer.

Yes i know but it just an example of the method but i write this so quickly normally i prefer writing echo "${a}",i use ls -la .. because i wanted to test my sshd_config (ChrootDirectory) but i rewrite this better :

#!/bin/bash +x

# with a varaiable or directly in file
# Replace "user", "host" and filename with your value
function list_sftp()
{
    list=`sftp user@host << EOL
    ls -la
EOL`
    echo "${list}" > tsftpres-000.txt
}

list_sftp

function other_lsftp()
{
    sftp user@host << EOL
    ls -la
EOL
}

other_lsftp > tsftpres-001.txt
1

The dir command within the sftp client does not support redirection. Example below, showing how it does nothing.

sftp> pwd
Remote working directory: /var/tmp/foodir
sftp> lcd /var/tmp/foodir
sftp> dir *.*
foo.txt
sftp> dir *.* >dirlist.txt
foo.txt
sftp> dir
foo.txt
sftp>

Man page for sftp confirms.

ls [-1afhlnrSt] [path]

    Display a remote directory listing of either path or the current directory if path is not specified.  path may contain glob(3) characters and may match multiple files.

    The following flags are recognized and alter the behaviour of ls accordingly:

    -1

        Produce single columnar output.
    -a
        List files beginning with a dot (‘.’).
    -f
        Do not sort the listing. The default sort order is lexicographical.
    -h
        When used with a long format option, use unit suffixes: Byte, Kilobyte, Megabyte, Gigabyte, Terabyte, Petabyte, and Exabyte in order to reduce the number of digits to four or fewer using powers of 2 for sizes (K=1024, M=1048576, etc.).
    -l
        Display additional details including permissions and ownership information.
    -n
        Produce a long listing with user and group information presented numerically.
    -r
        Reverse the sort order of the listing.
    -S
        Sort the listing by file size.
    -t
        Sort the listing by last modification time.

0

So you can use this method too (with different example) :

#!/bin/bash +x

fct()
{
        a=`sftp user@host<< EOL
        ls -la ..
EOL`
        echo $a
        echo $a > tsftpres.000
}

fct
fct > tsftpres.001

fct_2()
{
        sftp user@host << EOL
        ls -la ..
EOL
}

fct_2 > tsftpres.002
  • (1) I guess you’re trying to show variations on a theme, but this is a bit incoherent.  You demonstrate creating three files (one of them gets written twice).  Are they identical?  Is there any reason a user should prefer one of these commands over the others? (2) Why are you listing .. instead of remotelocation? (3) Have you tried this?  Does it really produce reasonable output?  I ask because I expect that echo "$a" would work much better than echo $a. (4) Does the name fct signify anything? – G-Man Says 'Reinstate Monica' Apr 25 at 16:28
2

This works by far the best:

echo 'ls' | sftp hostname

You can forward the output into a file by

echo 'ls' | sftp hostname > /tmp/mylist.txt
  • Returning one file per row should make the output easier to handle: echo 'ls -1a' | sftp hostname – dnlbrky May 6 at 21:22
0

You can also use lftp:

lftp -c 'connect sftp://user:password@host/dir; ls' > list.txt

Or

lftp -e 'ls;quit' sftp://user:password@host/dir > list.txt

lftp also has a find command for recursive listing.

Passing passwords on the command line is generally a bad idea. Alternative is to use ssh key authentication if possible (the password above will be ignored if key authentication is available, so you make it the empty string). Or you can pass it via the environment:

 LFTP_PASSWORD=password lftp --env-password -e 'ls;quit' \
   sftp://user:password@host/dir > file.list

Or pass the commands via stdin, here using a here document (implemented with deleted tempfiles or pipes depending on the shell):

lftp <<'EOF' > file.list
connect sftp://user:password@host/dir
ls
EOF
1

in case anyone finds rsync doesn't work (for reports.paypal.com in my case) I found the following script using expect works:

#!/usr/bin/expect
spawn sftp username@reports.paypal.com:/ppreports/outgoing
expect "password:"
send "XXXXXXXXXX\n"
expect "sftp>"
log_file -noappend RemoteFileList.txt
send "ls -1\n"
expect "sftp>"
log_file
send "!sed -i '' '/ls -1/d' ./RemoteFileList.txt\n"
expect "sftp>"
send "!sed -i '' '/sftp>/d' ./RemoteFileList.txt\n"
expect "sftp>"
send "bye\n"
interact

This is on Mac. On other systems the sed lines might do without the first '' arguments.

0

There is another way which I wound up using.. Try using rsync.

We find the following:

  • If you specify no local destination then a listing of the specified files on the remote server is provided.

The BASH script then becomes:

#!/bin/bash
localpath=/home/local-acct/path
remotelocation=/home/account/logs/archive
remotehost=' account@1.2.3.4'
rsync -avz $remotehost:/$remotelocation > $localpath/dirlist.txt
exit

and works great!

  • 2
    Note that rsync uses ssh (so assumes the remote user has shell access), not sftp (a subsystem of ssh that does not require the user having shell access) and assumes rsync is installed on the remote machine. So you might as well do ssh user@host ls /dir > file – Stéphane Chazelas Jun 9 '16 at 8:32
5

The sftp command is very limited. If you can't make it do what you want, you can use another approach, which is to mount the remote directory with the SSHFS filesystem. SSHFS uses SFTP as the transport protocol, so the server side just sees an SFTP client. You need to be able to use FUSE on the client side.

With SSHFS, you mount the remote directory onto an existing, empty directory and use ordinary commands.

mkdir remote
sshfs "$remotehost:$remotelocation" remote
cd remote

echo *.* >"$localpath/dirlist.txt"

fusermount -u remote
rmdir remote

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.