83

When I try to use sftp to transfer a directory containing files, I get an error message:

skipping non-regular file directory_name

The directory contains a couple of files and two subdirectories.

What am I doing wrong?

11 Answers 11

89

sftp, like cp and scp, requires that when you copy a folder (and its contents, obviously), you have to explicitly tell it you want to transfer the folder recursively with the -r option.

So, add -r to the command.

10
  • 2
    -r after the put command? I am getting a "put: Invalid flag -r" error message message if I do. Adding it to the original sftp command also does not work.
    – haziz
    Dec 17, 2011 at 9:12
  • 6
    Add -r to the sftp command when you're connecting. sftp -r user@host
    – Kevin
    Dec 17, 2011 at 15:50
  • 1
    @Kevin I am getting the following error "sftp: unknown option --r"
    – Amatya
    Mar 27, 2014 at 20:06
  • 5
    Beware that there is a quite long standing bug in OpenSSH's SFTP: when copying recursively a directory to a remote server, the directory itself must already exist on the target server otherwise the transfer will fail with a non-explicit error message: Couldn't canonicalise: No such file or directory. Dec 7, 2015 at 10:09
  • 9
    should be something like: # sftp user@remote-host #> get -r remote-folder
    – hbinduni
    Aug 1, 2017 at 5:03
33

I can only suggest, you use rsync. It is somewhat of an industry standard, when moving files over secure connections.

rsync -alPvz ./source_dir server.com:destination_dir

It is what I've been using for years by now.

(the -a option takes care of things like directory recursion)

3
  • 7
    A bit of clarification on the parameters mentioned: -a = recursion + keep symbolic links, permissions, modification times, owner, group, special files and device files (the latter requires super-user privileges); -l = keep symbolic links (already included in -a); -P = show progress during the transfer + keep partially transferred files; -v = verbose mode (however it seems mandatory when using -P to avoid a weird (bug?) protocol version mismatch error); -z = enable compression of transferred data. Dec 7, 2015 at 13:52
  • @WhiteWinterWolf it seems to be version dependant whether the -l switch has to be used or not. I remember older versions didn't include -l in -a.
    – polemon
    Dec 7, 2015 at 15:13
  • 3
    Using rsync is not adviced if you have very large number of files (40000+) as it might run out of memory - especially on smaller systems like Raspberry Pi 1-3. The more files you have, the more memory rsync uses (at least prior to version 3.1 - not sure if it's yet fixed in that or newer either), leading to rsync eventually using all system memory.
    – DocWeird
    Apr 16, 2020 at 10:16
17

This works for me:

1) connect via sftp to remote host

2) change into the remote directory you wish to copy. (Example: cd Music)

3) change to the local directory you wish to copy stuff to. (Example: lcd Desktop)

4) Issue this command: get -r *

0
10

You may also be able to use use scp. The general format is

scp -rp sourceDirName username@server:destDirName

scp means "secure copy". The flags are

  • -r recurse into subdirectories
  • -p preserve modification times

I believe the rest of the items are self-explanatory

2
  • 1
    and scp -rp -P <port> .... when using a non-standard port
    – Pancho
    May 7, 2016 at 9:25
  • I'd also add the -C flag for compression (both scp/sftp). btw - scp uses sftp under the hood since OpenSSH 9 - see also openssh.com/txt/release-9.0
    – Sandra
    Nov 10, 2022 at 15:48
6

If you question is actually 'how do I use sftp to transfer a directory', then

 sftp -r user@server

But if you transferring a directory, I might suggest two better options, depending on your needs. sftp requires logging into the remote server in a ssh-like command prompt, but if all you want is the file(s), there are easier and quicker ways.

Scp

For a one time transfer, to upload try:

 # upload source directory to remote server
 scp -rp source_dir user@server:dest

 #download directory from remote server:
 scp -rp user@server:source_dir dest

 #specifying ssh key:
 scp -rp -i ~/.ssh/key user@server:source_dir dest

 #remote directory has spaces
 scp -rp -i ~/.ssh/key user@server:"source\\ dir" dest

Rsync

If you plan on syncing the directories on a regular basis, using rsync makes more sense. It performs deltas between the two directories, saving transfer time and data over the wire.

rsync -r -a -v -e ssh --delete source_dir user@server:dest

Here is a list of some of the most common rsync options: (taken from here)

  • --delete - delete files that don't exist on sender (system)
  • -v - verbose (-vv will provide more detailed information)
  • -e "ssh options" - specify the ssh as remote shell
  • -a - archive mode - it preserves permissions (owners, groups), times, symbolic links, and devices
  • -r - recurse into directories
  • -z - compress file data during transfer
  • --exclude 'foldername' – excludes the corresponding folder from transfer
  • -P – show progress during transfer

Hope that helps!

5

If rsync is not an option, I would next recommend lftp:

lftp sftp://[email protected]/path/path/

To download recursively from server use:

mirror

To upload recursively to server use:

mirror -R

Note that recursion is the default behavior. The -R is for reverse — to make the mirror command upload instead of download. Use --dry-run to make sure it's doing what you expect.

To download without recursion use:

mirror --no-recursion

You have to cd into the directories you want to mirror. Works great!

0
3

If you can, use sshfs. It's a FUSE filesystem, available on most modern unices, and works with any SFTP server. This is a remote filesystem: it allows you to manipulate remote files (over the SFTP protocol) with the usual utilities.

mkdir /mount/point
sshfs server.example.com:/remote/path /mount/point
ls /mount/point
cp -Rp /mount/point/somedir /local/location
fusemount -d /mount/point
0

It's a bit of a kludge but what works for me is to:

  1. Use ssh to login to the remote machine
  2. Use sftp from the remote machine to your local machine
  3. Then use the get command with the -r option to move the directory and all of its files.
1
  • 1
    get dont work with -r sftp> get -r data /opt/bin get: Invalid flag -r
    – Junaid
    Aug 9, 2017 at 10:20
0

you can get from the server to your local path by

scp -rp user@server:directoryname(full path) .
2
0

It's a workaround,

  1. You need to connect via SFTP.
  2. Go to the directory where files reside.(Example: cd [remote directory name])
  3. If you have many files with the same format, you can use the get *.[file format]. Example: If you have many files with .csv format, then use,
get *.csv 
1
  • 1
    Welcome to this forum. Please explain how this answers the OP's question.
    – asoundmove
    Sep 28, 2020 at 21:32
0

I had the same need when I was already connected via sftp, and I didn't want to reconnect using -r param because in my case it's a bit annoying. There's a proxy which I have to connect first and then use a 2 factor token.

So, to avoid that, I did the following workaround: I compressed the folder using another ssh session and moved the resulting file on my sftp session.

SSH

tar -czvf myfolder.tar.gz myfolder

SFTP

put myfolder.tar.gz /server-path-name

Then, in the server (SSH), I extracted the folder by using:

tar -xzvf myfolder.tar.gz

If you're transferring a big amount of data, this workaround might be rather useful because it helps reducing the amount of data transferred and thus, the total transfer time.

1
  • I like your answer, as it makes it easier to see the total size, but does require space on the server to create the zip file. alternatively, you can use -C for compression with sftp, which would also reduce the amount transferred
    – Sandra
    Nov 10, 2022 at 15:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .