Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a large directory containing subdirectories and files that I wish to copy recursively.

Is there any way to tell cp that it should perform the copy operation in order of file size, so that the smallest files get copied first?

share|improve this question
    
Just to be sure there's not an XY problem involved, can you explain why you want to do this? –  goldilocks Apr 8 at 17:32
2  
@TAFKA'goldilocks' - I have a lot of video files, and I'd like to quality test each directory. The smallest video will give me a quick indication of if the rest of the files are bad as well. –  nbubis Apr 8 at 17:34
add comment

5 Answers 5

up vote 8 down vote accepted

This does the whole job in one go - in all child directories, all in a single stream without any filename problems. It'll copy from smallest to largest every file you have. You will need to mkdir ${DESTINATION} if it doesn't already exist.

find . ! -type d -print0 |
du -b0 --files0-from=/dev/stdin |
sort -zk1,1n | 
sed -zn 's/^[^0-9]*[0-9]*[^.]*//p' |
tar --hard-dereference --null -T /dev/stdin -cf - |
    tar -C"${DESTINATION}" --same-order -xvf -

You know what, though? What this doesn't do is empty child directories. I could do some redirection over that pipeline, but it's just a race condition waiting to happen. Simplest is probably best. So just do this afterwards:

find . -type d -printf 'mkdir -p "'"${DESTINATION}"'/%p"\n' |
    . /dev/stdin

Or, since Gilles makes a very good point in his answer to preserve directory permissions, I should try also. I think this will do it:

find . -type d -printf '[ -d "'"${DESTINATION}"'/%p" ] || 
    cp "%p" -t "'"${DESTINATION}"'"\n' |
. /dev/stdin

I'd be willing to bet that's faster than mkdir anyway.

share|improve this answer
    
Damn you mikeserv! +1 –  goldilocks Apr 8 at 20:53
    
@TAFKA'goldilocks' I'll take that as a compliment. Thanks very much. –  mikeserv Apr 8 at 20:58
add comment

Here is a quick and dirty method using rsync. For this example I am considering anything under 10 MB to be "small".

First transfer just the small files:

rsync -a --max-size=10m srcdir dstdir

Then transfer the remaining files. The previously transferred small files will not be re-copied unless they were modified.

rsync -a srcdir dstdir

From man 1 rsync

   --max-size=SIZE
          This  tells  rsync to avoid transferring any file that is larger
          than the specified SIZE. The SIZE value can be suffixed  with  a
          string  to  indicate  a size multiplier, and may be a fractional
          value (e.g. "--max-size=1.5m").

          This option is a transfer rule, not an exclude,  so  it  doesn’t
          affect  the  data  that  goes  into  the file-lists, and thus it
          doesn’t affect deletions.  It just limits  the  files  that  the
          receiver requests to be transferred.

          The  suffixes  are  as  follows:  "K"  (or  "KiB") is a kibibyte
          (1024), "M" (or "MiB") is a mebibyte (1024*1024),  and  "G"  (or
          "GiB")  is  a gibibyte (1024*1024*1024).  If you want the multi‐
          plier to be 1000 instead of  1024,  use  "KB",  "MB",  or  "GB".
          (Note: lower-case is also accepted for all values.)  Finally, if
          the suffix ends in either "+1" or "-1", the value will be offset
          by one byte in the indicated direction.

          Examples:    --max-size=1.5mb-1    is    1499999    bytes,   and
          --max-size=2g+1 is 2147483649 bytes.

Of course, the order of transfer file-by-file is not strictly smallest to largest, but I think it may be the simplest solution that meets the spirit of your requirements.

share|improve this answer
    
Here you get 2 copies of hard-links and soft-links are transformed into actual files for two copies of each. You'd do a lot better with --copy-dest=DIR and/or --compare-dest=DIR I think. I only know cause I had to add --hard-dereference myself to tar after posting my own answer because I was missing the links. I think rsync actually behaves more specific to local filesystems with those others anyway - I used to use it with USB keys and it would flood the bus unless I set a bandwidth limit. I think I should have used either of those others instead. –  mikeserv Apr 8 at 20:48
add comment

Not cp directly, that's well beyond its abilities. But you can arrange to call cp on the files in the right order.

Zsh conveniently allows sorting files by size with a glob qualifier. Here's a zsh snippet which copies files in increasing order of size from under /path/to/source-directory to under /path/to/destination-directory.

cd /path/to/source-directory
for x in **/*(.oL); do
  mkdir -p /path/to/destination-directory/$x:h
  cp $x /path/to/destination-directory/$x:h
done

Instead of a loop, you can use the zcp function. However you need to create the destination directories first, which can be done in a cryptic oneliner.

autoload -U zmv; alias zcp='zmv -C'
cd /path/to/source-directory
mkdir **/*(/e\''REPLY=/path/to/destination-directory/$REPLY'\')
zcp -Q '**/*(.oL)' '/path/to/destination-directory/$f'

This doesn't preserve the ownership of the source directories. If you want that, you'll need to enlist a suitable copying program such as cpio or pax. If you do that, you don't need to call cp or zcp in addition.

cd /path/to/source-directory
print -rN **/*(^.) **/*(.oL) | cpio -0 -p /path/to/destination-directory
share|improve this answer
add comment

I don't think there is any way to get cp -r to do this directly. Since it may be an indeterminate period of time before you get a wizardly find/awk solution, here's a quick perl script:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

use File::Find;
use File::Basename;

die "No (valid) source directory path given.\n"
    if (!$ARGV[0] || !-d -r "/$ARGV[0]");

die "No (valid) destination directory path given.\n"
    if (!$ARGV[1] || !-d -w "/$ARGV[1]");

my $len = length($ARGV[0]);
my @files;
find (
    sub {
        my $fpath = $File::Find::name;
        return if !-r -f $fpath;
        push @files, [
            substr($fpath, $len),
            (stat($fpath))[7],
        ]
    }, $ARGV[0]
);

foreach (sort { $a->[1] <=> $b->[1] } @files) {
    if ($ARGV[2]) {
        print "$_->[1] $ARGV[0]/$_->[0] -> $ARGV[1]/$_->[0]\n";
    } else {
        my $dest = "$ARGV[1]/$_->[0]";
        my $dir = dirname($dest);
        mkdir $dir if !-e $dir;
        `cp -a "$ARGV[0]/$_->[0]" $dest`;
    }
} 
  • Use this: ./whatever.pl /src/path /dest/path

  • The arguments should be both be absolute paths; ~, or anything else which the shell expands to an absolute path is fine.

  • If you add a third argument (anything, except for a literal 0), instead of copying it will print to standard out a report of what it would do, with files sizes in bytes prepended, e.g.

    4523 /src/path/file.x -> /dest/path/file.x
    12124 /src/path/file.z -> /dest/path/file.z
    

    Notice these are in ascending order by size.

  • The cp command on line 34 is a literal shell command, so you can do whatever you want with the switches (I just used -a to preserve all traits).

  • File::Find and File::Basename are both core modules, i.e. they are available in all installations of perl.

share|improve this answer
    
arguably, this is the only correct answer here. Or it was... the title - just changed...? My browser window is called cp - copy smallest files first? but the title of the post is just copy smallest files first? Anyway, options never hurt is my philosophy, but still, you and David are the only ones that used cp and you're the only one that pulled it off. –  mikeserv Apr 8 at 21:19
    
@mikeserv The only reason I used cp was because it's the simplest way to preserve *nix file characteristics in the (cross-platform oriented) perl. The reason your browser bar says cp - is because of a (IMO goofy) S.E. feature whereby the most popular of the selected tags appears prefixed to the actual title. –  goldilocks Apr 8 at 21:57
    
Ok, then I withdraw my compliment. Not really, you don't often see pearl coming out of the woodwork around here. –  mikeserv Apr 8 at 21:59
add comment

another option would be to use cp with the output from du:

oldIFS=$IFS
IFS=''
for i in $(du -sk *mpg | sort -n | cut -f 2)
do
    cp $i destination
done
IFS=$oldIFS

This could still be done on one line, but I split it so you can read it

share|improve this answer
    
Don't you at least need to do something about $IFS? –  mikeserv Apr 8 at 19:32
    
Yes...I keep assuming no one has newlines in their filenames –  David Wilkins Apr 8 at 19:33
1  
This also doesn't seem to handle recursion through the directory hierarchy the OP described. –  cpugeniusmv Apr 8 at 19:39
1  
@cpugeniusmv Correct...I somehow missed the recursive part....I could modify this to handle recursion, but I think at this point other answers do a better job. I'll leave this here in case it helps someone who sees the question. –  David Wilkins Apr 8 at 19:41
1  
@DavidWilkins - this helps a lot. –  nbubis Apr 8 at 22:31
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.