86

I want to use scp to upload files but sometimes the target directory may not exist.

Is it possible to create the folder automatically? If so, how? If not, what alternative way can I try?

69

This is one of the many things that rsync can do.

If you're using a version of rsync released in the past several years,¹ its basic command syntax is similar to scp

$ rsync -r local-dir remote-machine:path

That will copy local-source and its contents to $HOME/path/local-dir on the remote machine, creating whatever directories are required.³

rsync does have some restrictions here that can affect whether this will work in your particular situation. It won't create multiple levels of missing remote directories, for example; it will only create up to one missing level on the remote. You can easily get around this by preceding the rsync command with something like this:

$ ssh remote-host 'mkdir -p foo/bar/qux'

That will create the $HOME/foo/bar/qux tree if it doesn't exist. It won't complain or do anything else bad if it does already exist.

rsync sometimes has other surprising behaviors. Basically, you're asking it to figure out what you meant to copy, and its guesses may not match your assumptions. Try it and see. If it doesn't behave as you expect and you can't see why, post more details about your local and remote directory setups, and give the command you tried.


Footnotes:

  1. Before rsync 2.6.0 (1 Jan 2004), it required the -e ssh flag to make it behave like scp because it defaulted to the obsolete RSH protocol.

  2. scp and rsync share some flags, but there is only a bit of overlap.

  3. When using SSH as the transfer protocol, rsync uses the same defaults. So, just like scp, it will assume there is a user with the same name as your local user on the remote machine by default.

  • I am using Mac OSX, do I need to install rsync on both local and remote side? – AGamePlayer Mar 30 '15 at 13:27
  • searched rsync's documentation page, did not find how to use ssh-key to login. I don't want to use username/password to login – AGamePlayer Mar 30 '15 at 13:28
  • @AwQiruiGuo: Mac OS X ships with rsync. And yes, you do need rsync on the remote side, too, just like with scp. As for SSH keys, rsync is actually using SSH underneath with the default -e ssh, so if you have keys set up for use with scp, they will work for rsync as well. – Warren Young Mar 30 '15 at 13:29
  • 1
    @WarrenYoung -r makes no difference. The scenario you describe only works if you are copying a directory and if the target parent directory exists. So, rsync foo/ u@h:~/ will create the target directory foo but rsync foo/ u@h:~/bar/ will not create the target directory bar. That one will create bar/foo only if bar/ exists. In any case, the OP is uploading files and even in the cases where this works, it only works for entire directories. – terdon Mar 30 '15 at 13:29
  • 1
    Yes, I've been playing with it too. It seems like specifying a file name also makes it fail. So rsync file host:/foo/file fails if foo/ doesn't exist but rsync file host:/foo/ creates the directory and places file in it. Strange. Anyway, you're quite right and the basic scenario works, I'll retract my downvote. – terdon Mar 30 '15 at 13:55
34

If you are copying a group of files, not really. If you are copying a directory and all the contents below it, yes. Given this command:

$ scp -pr /source/directory user@host:the/target/directory

If directory does not exist in ~/the/target on host, it will be created. If, however, ~the/target does not exist, you will likely have issues - I don't remember the exact results of this case, but be prepared for the intended scp to fail.

  • 1
    scp: /home/madmike/test_bin: No such file or directory As soon as the target directory is missing this breaks on my setup. Using cygwin on the client and RHEL 6.9 on the server. What is your setup? – MadMike Feb 12 '18 at 15:00
16

As far as I know, scp itself cannot do that, no. However, you could just ssh to the target machine, create the directory and then copy. Something like:

ssh user@host "mkdir -p /target/path/" &&
    scp /path/to/source user@host:/target/path/

Note that if you are copying entire directories, the above is not needed. For example, to copy the directory ~/foo to the remote host, you could use the -r (recursive) flag:

scp ~/foo/ user@host:~/

That will create the target directory ~/foo on the remote host. However, it can't create the parent directory. scp ~/foo user@host:~/bar/foo will fail unless the target directory bar exists. In any case, the -r flag won't help create a target directory if you are copying individual files.

3

Yes, you can. According to scp's man page:

man scp

.....

-r Recursively copy entire directories. Note that scp follows symâ bolic links encountered in the tree traversal.

....

  • 6
    That will not create the target parent directory. It can copy a directory to an existing location. So, scp -r foo/ u@h:~/bar will create bar/foo if the remote bar/ exists but will fail if it doesn't. – terdon Mar 30 '15 at 13:32
  • 1
    It won't fail. It will copy foo contents to the new directory /bar. Of course the name has been changed which may not be what the OP wants. If what you want is a new tree structure created in the destination machine then scp won't do the job, if what you want is to replicate a tree structure from a certain branch in the tree then that structure is created. – YoMismo Mar 30 '15 at 13:46
  • 6
    Yes, but this only works if you are copying directories, not files. scp -r file host:~/bar/ won't create bar/. – terdon Mar 30 '15 at 13:57
2

I use the following function before transferring btrfs snapshots over ssh:

check_remote_dir() {
        printf "\ntesting remote directory: '$1' "

        if ssh -p $PORT $ROOT@$REMOTE "[ ! -d $1 ]"; then
                printf "\nCreating: $1 on $ROOT@$REMOTE\n"
                ssh -p $PORT $ROOT@$REMOTE "mkdir -p $1"
        else
                printf "[OK]\n"
        fi
}

Just call the function in your script with:

check_remote_dir /my/remote/path

  • I do pretty much the same thing, but I have to type in the password manually (for security reasons), so I end up typing it in at least twice... – Stefan Reich Dec 14 '18 at 12:25
0

You could chain the command with the mkdir command and then reference to the folder you've created:

mkdir ~/new-folder/ && scp -P 22 <remote/url>:~/new-folder/
  • 3
    (1) The question says “automatically”.   This does not appear to be automatic.   (2) This appears to be only a trivial variation on terdon’s answer. – G-Man Jan 22 at 9:50
0

When restricted to only scp or sftp with no ssh available (by rssh), it is possible to create directories by copying empty directory tree with rsync and then copying files.

That is how I put public ssh key to remote host when ssh-copy-id does not work and .ssh directory does not exists:

rsync -e 'ssh -p22' -av -f"+ */" -f"- *" ~/.ssh backup@1.2.3.4:~/
scp -P22 ~/.ssh/id_rsa.pub backup@1.2.3.4:~/.ssh/authorized_keys

It is possible to combine this commands into one rsync adding one more include filter in the middle, if target file name is the same.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.