Can scp create a directory if it doesn't exist?

Question

I want to use scp to upload files but sometimes the target directory may not exist.

Is it possible to create the folder automatically? If so, how? If not, what alternative way can I try?

Answer 1

This is one of the many things that rsync can do.

If you're using a version of rsync released in the past several years,¹ its basic command syntax is similar to scp:²

$ rsync -r local-dir remote-machine:path

That will copy local-source and its contents to $HOME/path/local-dir on the remote machine, creating whatever directories are required.³

rsync does have some restrictions here that can affect whether this will work in your particular situation. It won't create multiple levels of missing remote directories, for example; it will only create up to one missing level on the remote. You can easily get around this by preceding the rsync command with something like this:

$ ssh remote-host 'mkdir -p foo/bar/qux'

That will create the $HOME/foo/bar/qux tree if it doesn't exist. It won't complain or do anything else bad if it does already exist.

rsync sometimes has other surprising behaviors. Basically, you're asking it to figure out what you meant to copy, and its guesses may not match your assumptions. Try it and see. If it doesn't behave as you expect and you can't see why, post more details about your local and remote directory setups, and give the command you tried.

---

Footnotes:

1. Before rsync 2.6.0 (1 Jan 2004), it required the -e ssh flag to make it behave like scp because it defaulted to the obsolete RSH protocol.

2. scp and rsync share some flags, but there is only a bit of overlap.

3. When using SSH as the transfer protocol, rsync uses the same defaults. So, just like scp, it will assume there is a user with the same name as your local user on the remote machine by default.

Answer 2

If you are copying a group of files, not really. If you are copying a directory and all the contents below it, yes. Given this command:

$ scp -pr /source/directory user@host:the/target/directory

If directory does not exist in ~/the/target on host, it will be created. If, however, ~the/target does not exist, you will likely have issues - I don't remember the exact results of this case, but be prepared for the intended scp to fail.

Answer 3

As far as I know, scp itself cannot do that, no. However, you could just ssh to the target machine, create the directory and then copy. Something like:

ssh user@host "mkdir -p /target/path/" &&
    scp /path/to/source user@host:/target/path/

Note that if you are copying entire directories, the above is not needed. For example, to copy the directory ~/foo to the remote host, you could use the -r (recursive) flag:

scp -r ~/foo/ user@host:~/

That will create the target directory ~/foo on the remote host. However, it can't create the parent directory. scp ~/foo user@host:~/bar/foo will fail unless the target directory bar exists. In any case, the -r flag won't help create a target directory if you are copying individual files.

Answer 4

I use the following function before transferring btrfs snapshots over ssh:

check_remote_dir() {
        printf "\ntesting remote directory: '$1' "

        if ssh -p $PORT $ROOT@$REMOTE "[ ! -d $1 ]"; then
                printf "\nCreating: $1 on $ROOT@$REMOTE\n"
                ssh -p $PORT $ROOT@$REMOTE "mkdir -p $1"
        else
                printf "[OK]\n"
        fi
}

Just call the function in your script with:

check_remote_dir /my/remote/path

Answer 5

Yes, you can. According to scp's man page:

man scp

.....

-r Recursively copy entire directories. Note that scp follows symâ bolic links encountered in the tree traversal.

....

Answer 6

When restricted to only scp or sftp with no ssh available (by rssh), it is possible to create directories by copying empty directory tree with rsync and then copying files.

That is how I put public ssh key to remote host when ssh-copy-id does not work and .ssh directory does not exists:

rsync -e 'ssh -p22' -av -f"+ */" -f"- *" ~/.ssh [email protected]:~/
scp -P22 ~/.ssh/id_rsa.pub [email protected]:~/.ssh/authorized_keys

It is possible to combine this commands into one rsync adding one more include filter in the middle, if target file name is the same.

Answer 7

How about "smart" scp + ssh mkdir?

This is a script to create target directory hierarchy only once per directory (not a blind ssh mkdir -p each time).

Rationale

I'm copying motion recordings (video files) as they are written to local disk (event-driven).

• rsync doesn't make sense because there is no point in doing full directory comparisons - I am transferring one file per event. If I restrict rsync to just one subdir or file, then it won't be able to create the full target directory hierarchy.
• scp will work great but it doesn't know if the target directory exists, and I don't want to run an ssh mkdir -p for every file.

Solution

1. Create each target dir via ssh the first time there is a need (first time we are about to copy into it).
2. Cache target dir name in a an array so that we won't ssh mkdir each time we see it again. Clear this array daily so that it doesn't get huge.
3. scp the file.

Example code:

#!/bin/bash

TARGETHOST="some.hostname"
TARGETPATH="/data/backups/MOTION/"

ensure_dir() {
    if [ "$TODAY" != "$(date +%D)" ] ; then
        CREATED=()  # reset array daily and first time so it won't get huge
        TODAY=$(date +%D)
    fi  
    if [[ ! " ${CREATED[@]} " =~ " ${1} " ]]; then
        # create dir
        echo "Creating remote directory: $1"
        ssh $TARGETHOST "mkdir -p ${TARGETPATH}$1"
        CREATED+=("$1")
    fi  
}

# This line is specific to my use case, watching a directory for new files:
inotifywait -mr --format '%w %f' -e close_write --exclude '.jpg' /data/MOTION/ | while read filedir filename ; do
    RELDIR=$(echo $filedir | sed 's|/data/MOTION/||')
    ensure_dir "$RELDIR"
    echo "Copying '$filename' from '${filedir}' to '${RELDIR}'"
    scp -r -p "${filedir}${filename}" "${TARGETHOST}:${TARGETPATH}${RELDIR}${filename}"
done

Typical output, illustrating "smart" mkdir behavior:

Creating remote directory: 2019-12-28/movie-C6/
Copying 'test1.mp4' from '/data/MOTION/2019-12-28/movie-C6/' to '2019-12-28/movie-C6/'
Creating remote directory: 2019-12-28/movie-C4/
Copying '095700.mp4' from '/data/MOTION/2019-12-28/movie-C4/' to '2019-12-28/movie-C4/'
Copying '095841.mp4' from '/data/MOTION/2019-12-28/movie-C4/' to '2019-12-28/movie-C4/'
Copying '100410.mp4' from '/data/MOTION/2019-12-28/movie-C4/' to '2019-12-28/movie-C4/'
Creating remote directory: 2019-12-28/movie-C2/
Copying '102106.mp4' from '/data/MOTION/2019-12-28/movie-C2/' to '2019-12-28/movie-C2/'
Creating remote directory: 2019-12-28/movie-C5/
Copying '102318.mp4' from '/data/MOTION/2019-12-28/movie-C5/' to '2019-12-28/movie-C5/'
Copying '102326.mp4' from '/data/MOTION/2019-12-28/movie-C2/' to '2019-12-28/movie-C2/'
Creating remote directory: 2019-12-28/movie-C1/
Copying '102450.mp4' from '/data/MOTION/2019-12-28/movie-C1/' to '2019-12-28/movie-C1/'
Copying '102744.mp4' from '/data/MOTION/2019-12-28/movie-C2/' to '2019-12-28/movie-C2/'
Copying '103008.mp4' from '/data/MOTION/2019-12-28/movie-C2/' to '2019-12-28/movie-C2/'
Copying '103945.mp4' from '/data/MOTION/2019-12-28/movie-C6/' to '2019-12-28/movie-C6/'
Copying '104252.mp4' from '/data/MOTION/2019-12-28/movie-C6/' to '2019-12-28/movie-C6/'
Copying '104824.mp4' from '/data/MOTION/2019-12-28/movie-C6/' to '2019-12-28/movie-C6/'

Answer 8

You could chain the command with the mkdir command and then reference to the folder you've created:

mkdir ~/new-folder/ && scp -P 22 <remote/url>:~/new-folder/