I'm looking for a command/script to allow the most recently modified files (up to) 10GB be copied to another computer.

So if there are 4 files 4 GB each, only 2 of them should be transferred by the script, If there are 12 files 1GB big, only the most recent 10 should be transferred.

  • 1
    I can't think of any way to do this, but to clarify your question, do you really want the most recently modified 10GB of files copied, or any set of up to 10GB files? I don't believe there's any way to force rsync to give priority to the most recent files. The closest answer I can think of would be to constrain bandwidth to a known value (like 1MB/second) and kill rsync after enough time has elapsed to transfer x GB of data. Not perfect since the bandwidth constraint is a maximum value so you may not transfer as much as you wanted.
    – Johnny
    Oct 23, 2013 at 0:40
  • the most recent. by file mtime
    – exussum
    Oct 23, 2013 at 7:40

3 Answers 3


Here is a script that does just what you asked for.

The requirements

  • The files transferred must total less than a threshold size.
  • The files must be modified compared to the rsync destination.
  • If not all files can be transferred, only the most recently modified files must be selected.

The details

It uses rsync --dry-run to build a list of files that would be transferred (these are the modified files). It then uses a combination of du and ls to get file sizes and mtime. It then sorts the files by mtime and then loops over them until the total size exceeds a threshold. Finally, it calls rsync again with just the files that are the most recently modified and total size under the threshold.

The script is a bit ugly, but it works. One big limitation is that it must be executed on the machine containing the rsync from-directory. It can be modified to use ssh to use a remote from-directory, but that excersize is left to the reader.

Finally, the rsync options are hard coded into the script, but this is an easy change if you want to specify them on the command line. Also, the math to calculate size is done in bytes. This can be changed to kilo/mega/gigabytes by modifying the call to du and reducing the threshold by the same factor.


./rsyncrecent.sh rsync-from-directory rsync-to-directory

where rsync-from-directory is a local directory and rsync-to-directory is any local or remote directory. The default options are hardcoded as -avz and the default threshold is hardcoded as 10GiB.

The script



usage () {
  echo >&2 "Usage:  $0 from-location to-location"
  exit 1

[ "$#" -eq 2 ] || usage


echo "Fetching file list for $RSYNC $RSYNC_OPTS $RSYNC_FROM $RSYNC_TO"

# get list of changed files
FILES=`$RSYNC $RSYNC_OPTS --dry-run  $RSYNC_FROM $RSYNC_TO | sed -n '/list$/,/^$/{/sending.*list$/ d ; /^$/ d ; /\/$/ d ;; p}'`

# reported files are relative to ..RSYNC_FROM, so rather than transforming filenames, lets just move there
pushd $RSYNC_FROM > /dev/null

# get modified time and sizes for all files
for FILE in $FILES
   #strip first part of path so files are relative to RSYNC_FROM
   #FSIZE=`ls -l $FILE | cut -f5 -d' '`
   FSIZE=`du -bs $FILE`
   FMTIME=`ls -l --time-style=+%s $FILE | cut -f6 -d' '`
   FLIST[$i]=`echo $FMTIME $FILE $FSIZE`

# go back to original directory
popd > /dev/null

# sort list according to modified time
IFS=$'\n' FLIST=($(sort -rg <<<"${FLIST[*]}"))


# add up the files in mtime order until threshold is reached
for ((i=0; i<$max; i++))
   s=`echo ${FLIST[$i]} | cut -f3 -d' '`
   f=`echo ${FLIST[$i]} | cut -f2 -d' '`
   if (( "$size" > "$THRESHOLD" ))
   echo $f >> /tmp/rsyncfilelist

$RSYNC $RSYNC_OPTS --dry-run $RSYNC_FROM --files-from=/tmp/rsyncfilelist  $RSYNC_TO

rm /tmp/rsyncfilelist
  • Works great, One time it doesnt work is when there is a file larger than 10GB as the most recent file
    – exussum
    Dec 5, 2013 at 21:16
  • If you always want the first file to transfer regardless of the threshold, in the final loop inside the if (( "$size" > "$THRESHOLD" )) conditional add a check (before break) for i==0 and if so, echo $f >> /tmp/rsyncfilelist.
    – casey
    Dec 5, 2013 at 23:04

I would use rsync "--dry-run" (or "-n") to get the list of the newer files. Then I would use another rsync with option "--files-from=-" to send the files. In between there is "ugly" perl.
Something like this :


$maxsize=10*1024**3; # 10GB 

open (RSOUT,"|rsync -av --files-from=- $source $target");
open (RSIN, "rsync -avn $source $target |");
while (<RSIN>)
        last if (/^$/);
        if (-f "$_")
                next if ($size + -s "$_" > $maxsize);
                $size += -s "$_";
                printf RSOUT "%s\n", $_;

Note I didn't test with more than 10GB, maybe perl will overflow at some limit; to solve that, instead of counting bytes use Kbytes :

$maxsize=10*1024**2; # 10M of Kbytes
     $size +=( -s "$_")/1024;

EDIT: I noted that this first solution would not sort file by mtime, here is a more complete solution (similar to the bash script that has been posted by another person).

use File::stat;

$maxsize=10 * 1024**3; # 10GB  

open (RSOUT,"|rsync -av --files-from=- $source $target");
open (RSIN, "rsync -avn $source $target |");
while (<RSIN>)
    last if (/^$/);
    if (-f "$_")
            my $fileattr;
            my $stat=stat($_);
            $hash{sprintf ("%s %s\n", $stat->mtime, $_)}=$fileattr;


foreach $key (reverse sort keys %hash)
    next if ( ($size + $hash{$key}->{size}) > $maxsize);
    $size += $hash{$key}->{size};
    print RSOUT $hash{$key}->{name}, "\n";

You can parse the sorted output of du. Assuming GNU utilities:

du -0ak | sort -z -k1n | awk -v 'RS=\0' -v 'ORS=\0' '
    (size += $1) > 10*1024*1024 {quit}
    {print substr($0, index(s, "\t")+1)}
' | xargs -0 cp -t destination

POSIXly, assuming that no file name contains a newline character:

du -ak | sort -k1n | awk '
    (size += $1) > 10*1024*1024 {quit}
    {print substr($0, index(s, "\t")+1)}
' | while IFS= read -r filename; do cp -- "$filename" /path/to/destination

Note that du traverses subdirectories. To avoid that, tell du which files you want to operate on. More generally, you can use find to filter files.

find . -type f ! -name excluded-file -exec du -ak {} + |
sort -k1n | awk '
    (size += $1) > 10*1024*1024 {quit}
    {print substr($0, index(s, "\t")+1)}
' | while IFS= read -r filename; do cp -- "$filename" /path/to/destination
  • is there a way to add rsync like functions ? this will be run more than once but this script will copy the files multiple times ?
    – exussum
    Oct 23, 2013 at 19:21
  • @user1281385 You can call rsync instead of cp. Oct 23, 2013 at 22:52
  • the rysnc function would be to remove the old ones when run multiple times not to not transfer the file if already exists
    – exussum
    Nov 28, 2013 at 17:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .