copy smallest files first?

Question

I have a large directory containing subdirectories and files that I wish to copy recursively.

Is there any way to tell cp that it should perform the copy operation in order of file size, so that the smallest files get copied first?

Answer 1

Here is a quick and dirty method using rsync. For this example I am considering anything under 10 MB to be "small".

First transfer just the small files:

rsync -a --max-size=10m srcdir dstdir

Then transfer the remaining files. The previously transferred small files will not be re-copied unless they were modified.

rsync -a srcdir dstdir

From man 1 rsync

   --max-size=SIZE
          This  tells  rsync to avoid transferring any file that is larger
          than the specified SIZE. The SIZE value can be suffixed  with  a
          string  to  indicate  a size multiplier, and may be a fractional
          value (e.g. "--max-size=1.5m").

          This option is a transfer rule, not an exclude,  so  it  doesn’t
          affect  the  data  that  goes  into  the file-lists, and thus it
          doesn’t affect deletions.  It just limits  the  files  that  the
          receiver requests to be transferred.

          The  suffixes  are  as  follows:  "K"  (or  "KiB") is a kibibyte
          (1024), "M" (or "MiB") is a mebibyte (1024*1024),  and  "G"  (or
          "GiB")  is  a gibibyte (1024*1024*1024).  If you want the multi‐
          plier to be 1000 instead of  1024,  use  "KB",  "MB",  or  "GB".
          (Note: lower-case is also accepted for all values.)  Finally, if
          the suffix ends in either "+1" or "-1", the value will be offset
          by one byte in the indicated direction.

          Examples:    --max-size=1.5mb-1    is    1499999    bytes,   and
          --max-size=2g+1 is 2147483649 bytes.

Of course, the order of transfer file-by-file is not strictly smallest to largest, but I think it may be the simplest solution that meets the spirit of your requirements.

Answer 2

This does the whole job in one go - in all child directories, all in a single stream without any filename problems. It'll copy from smallest to largest every file you have. You will need to mkdir ${DESTINATION} if it doesn't already exist.

find . ! -type d -print0 |
du -b0 --files0-from=/dev/stdin |
sort -zk1,1n | 
sed -zn 's/^[^0-9]*[0-9]*[^.]*//p' |
tar --hard-dereference --null -T /dev/stdin -cf - |
    tar -C"${DESTINATION}" --same-order -xvf -

You know what, though? What this doesn't do is empty child directories. I could do some redirection over that pipeline, but it's just a race condition waiting to happen. Simplest is probably best. So just do this afterwards:

find . -type d -printf 'mkdir -p "'"${DESTINATION}"'/%p"\n' |
    . /dev/stdin

Or, since Gilles makes a very good point in his answer to preserve directory permissions, I should try also. I think this will do it:

find . -type d -printf '[ -d "'"${DESTINATION}"'/%p" ] || 
    cp "%p" -t "'"${DESTINATION}"'"\n' |
. /dev/stdin

I'd be willing to bet that's faster than mkdir anyway.

Answer 3

Not cp directly, that's well beyond its abilities. But you can arrange to call cp on the files in the right order.

Zsh conveniently allows sorting files by size with a glob qualifier. Here's a zsh snippet which copies files in increasing order of size from under /path/to/source-directory to under /path/to/destination-directory.

cd /path/to/source-directory
for x in **/*(.oL); do
  mkdir -p /path/to/destination-directory/$x:h &&
    cp -- $x /path/to/destination-directory/$x:h
done

Instead of a loop, you can use the zcp function. However you need to create the destination directories first, which can be done in a cryptic oneliner.

autoload -U zmv; alias zcp='zmv -C'
cd /path/to/source-directory
mkdir -p **/*(/e\''REPLY=/path/to/destination-directory/$REPLY'\')
zcp -Q '**/*(.oL)' '/path/to/destination-directory/$f'

This doesn't preserve the ownership of the source directories. If you want that, you'll need to enlist a suitable copying program such as cpio or pax. If you do that, you don't need to call cp or zcp in addition.

cd /path/to/source-directory
print -rN -- **/*(^.) **/*(.oL) | cpio -0 -p /path/to/destination-directory

Answer 4

Here I'm using a --max-size option for rsync, but doing it in steps of 1 MB and then the other (larger) files, with this rsync proxy script:

#!/bin/bash
for i in {1..1024}
do
        echo Max Size $i\M
        rsync --max-size=$i\M "$@"
done
rsync "$@"

Answer 5

I don't think there is any way to get cp -r to do this directly. Since it may be an indeterminate period of time before you get a wizardly find/awk solution, here's a quick perl script:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

use File::Find;
use File::Basename;

die "No (valid) source directory path given.\n"
    if (!$ARGV[0] || !-d -r "/$ARGV[0]");

die "No (valid) destination directory path given.\n"
    if (!$ARGV[1] || !-d -w "/$ARGV[1]");

my $len = length($ARGV[0]);
my @files;
find (
    sub {
        my $fpath = $File::Find::name;
        return if !-r -f $fpath;
        push @files, [
            substr($fpath, $len),
            (stat($fpath))[7],
        ]
    }, $ARGV[0]
);

foreach (sort { $a->[1] <=> $b->[1] } @files) {
    if ($ARGV[2]) {
        print "$_->[1] $ARGV[0]/$_->[0] -> $ARGV[1]/$_->[0]\n";
    } else {
        my $dest = "$ARGV[1]/$_->[0]";
        my $dir = dirname($dest);
        mkdir $dir if !-e $dir;
        `cp -a "$ARGV[0]/$_->[0]" $dest`;
    }
} 

• Use this: ./whatever.pl /src/path /dest/path

• The arguments should be both be absolute paths; ~, or anything else which the shell expands to an absolute path is fine.

• If you add a third argument (anything, except for a literal 0), instead of copying it will print to standard out a report of what it would do, with files sizes in bytes prepended, e.g.

  4523 /src/path/file.x -> /dest/path/file.x
  12124 /src/path/file.z -> /dest/path/file.z
  

  Notice these are in ascending order by size.

• The cp command on line 34 is a literal shell command, so you can do whatever you want with the switches (I just used -a to preserve all traits).

• File::Find and File::Basename are both core modules, i.e. they are available in all installations of perl.

Answer 6

another option would be to use cp with the output from du:

oldIFS=$IFS
IFS=''
for i in $(du -sk *mpg | sort -n | cut -f 2)
do
    cp $i destination
done
IFS=$oldIFS

This could still be done on one line, but I split it so you can read it