SCP from a distant server with a cumulative size limit

Question

I want to copy a number of files from a distant server to local computer, such that the total size of the files does not exceed certain limit, as I do not locally have enough disk space; and the files are interchangeable samples of a greater whole, i.e. it does not matter which individual files are downloaded and in which order, just that a number of such files is downloaded within the disk space limit.

The files are downloaded as normal, but the download stops once the size limit is breached and the currently processed file is discarded.

Does SCP have such capability? I know that RSYNC can limit file size, but that is for individual files, not for cumulative file size.

If SCP nor RSYNC cannot do that, is there any other way?

Answer 1

There is nothing in either scp or rsync that will address the specific requirement. However, it seems to me that a standard loop-until-done approach could work

#!/bin/bash

# A set of candidate files to transfer. The construct here assumes
# straightforward filename (no spaces or other strangeness)
files=( $(ssh -n remoteHost 'echo *.dat') )

cd /dest/dir

maxSize=1024                                       # MB allowance
startUsed=$(df -m . | awk 'NR==2 {print $3+0}')    # MB used at start
st=                                                # Status stop reason

for file in "${files[@]}"
do
    # Enough disk space
    thisUsed=$(df -m . | awk 'NR==2 {print $3+0}')
    [ $(( thisUsed - startUsed )) -ge $maxSize ] && st=used && break

    # Copy the next selected file
    scp -p remoteHost:"$file" .        # scp (using sftp)
    # scp -Op remoteHost:"$file" .     # scp (using scp)
    # rsync -t remoteHost:"$file" .    # rsync

    [ $? -gt 0 ] && st=copy && break
done

# All done
case "$st" in
    used)    echo "INFO: Allowance reached" >&2 ;;
    copy)    echo "ERROR: Copy failed: $file" >&2 ;;
esac

The only dependency on bash (rather than POSIX sh) is the list/array of files at the beginning. If you can generate the list of files in some other way you can dispense with the dependency.