Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm looking for a command/script to allow the most recently modified files (up to) 10GB be copied to another computer.

So if there are 4 files 4 GB each, only 2 of them should be transferred by the script, If there are 12 files 1GB big, only the most recent 10 should be transferred.

share|improve this question
1  
I can't think of any way to do this, but to clarify your question, do you really want the most recently modified 10GB of files copied, or any set of up to 10GB files? I don't believe there's any way to force rsync to give priority to the most recent files. The closest answer I can think of would be to constrain bandwidth to a known value (like 1MB/second) and kill rsync after enough time has elapsed to transfer x GB of data. Not perfect since the bandwidth constraint is a maximum value so you may not transfer as much as you wanted. –  Johnny Oct 23 '13 at 0:40
    
the most recent. by file mtime –  exussum Oct 23 '13 at 7:40

3 Answers 3

up vote 3 down vote accepted
+50

Here is a script that does just what you asked for.

The requirements

  • The files transferred must total less than a threshold size.
  • The files must be modified compared to the rsync destination.
  • If not all files can be transferred, only the most recently modified files must be selected.

The details

It uses rsync --dry-run to build a list of files that would be transferred (these are the modified files). It then uses a combination of du and ls to get file sizes and mtime. It then sorts the files by mtime and then loops over them until the total size exceeds a threshold. Finally, it calls rsync again with just the files that are the most recently modified and total size under the threshold.

The script is a bit ugly, but it works. One big limitation is that it must be executed on the machine containing the rsync from-directory. It can be modified to use ssh to use a remote from-directory, but that excersize is left to the reader.

Finally, the rsync options are hard coded into the script, but this is an easy change if you want to specify them on the command line. Also, the math to calculate size is done in bytes. This can be changed to kilo/mega/gigabytes by modifying the call to du and reducing the threshold by the same factor.

Usage

./rsyncrecent.sh rsync-from-directory rsync-to-directory

where rsync-from-directory is a local directory and rsync-to-directory is any local or remote directory. The default options are hardcoded as -avz and the default threshold is hardcoded as 10GiB.

The script

#!/bin/bash

RSYNC=rsync
RSYNC_OPTS=-avz
THRESHOLD=10737418240

usage () {
  echo >&2 "Usage:  $0 from-location to-location"
  exit 1
}

[ "$#" -eq 2 ] || usage

RSYNC_FROM=$1
RSYNC_TO=$2

echo "Fetching file list for $RSYNC $RSYNC_OPTS $RSYNC_FROM $RSYNC_TO"

# get list of changed files
FILES=`$RSYNC $RSYNC_OPTS --dry-run  $RSYNC_FROM $RSYNC_TO | sed -n '/list$/,/^$/{/sending.*list$/ d ; /^$/ d ; /\/$/ d ;; p}'`

# reported files are relative to ..RSYNC_FROM, so rather than transforming filenames, lets just move there
pushd $RSYNC_FROM > /dev/null

# get modified time and sizes for all files
i=0
for FILE in $FILES
do
   #strip first part of path so files are relative to RSYNC_FROM
   FILE=${FILE#*/}
   #FSIZE=`ls -l $FILE | cut -f5 -d' '`
   FSIZE=`du -bs $FILE`
   FMTIME=`ls -l --time-style=+%s $FILE | cut -f6 -d' '`
   FLIST[$i]=`echo $FMTIME $FILE $FSIZE`
   ((i=$i+1))
done

# go back to original directory
popd > /dev/null

# sort list according to modified time
IFS=$'\n' FLIST=($(sort -rg <<<"${FLIST[*]}"))

max=$i
i=0
size=0
#NEWFLIST=''

# add up the files in mtime order until threshold is reached
for ((i=0; i<$max; i++))
do
   s=`echo ${FLIST[$i]} | cut -f3 -d' '`
   f=`echo ${FLIST[$i]} | cut -f2 -d' '`
   ((size=$size+$s))
   if (( "$size" > "$THRESHOLD" ))
   then
      break
   fi
   NEWFLIST="$NEWFLIST $f"
   echo $f >> /tmp/rsyncfilelist
done

$RSYNC $RSYNC_OPTS --dry-run $RSYNC_FROM --files-from=/tmp/rsyncfilelist  $RSYNC_TO

rm /tmp/rsyncfilelist
share|improve this answer
    
Works great, One time it doesnt work is when there is a file larger than 10GB as the most recent file –  exussum Dec 5 '13 at 21:16
    
If you always want the first file to transfer regardless of the threshold, in the final loop inside the if (( "$size" > "$THRESHOLD" )) conditional add a check (before break) for i==0 and if so, echo $f >> /tmp/rsyncfilelist. –  casey Dec 5 '13 at 23:04

I would use rsync "--dry-run" (or "-n") to get the list of the newer files. Then I would use another rsync with option "--files-from=-" to send the files. In between there is "ugly" perl.
Something like this :

#!/usr/bin/perl

$source="/somedir";
$target="host:/remotedir";
$maxsize=10*1024**3; # 10GB 

open (RSOUT,"|rsync -av --files-from=- $source $target");
open (RSIN, "rsync -avn $source $target |");
while (<RSIN>)
{
        chomp;
        last if (/^$/);
        if (-f "$_")
        {
                next if ($size + -s "$_" > $maxsize);
                $size += -s "$_";
                printf RSOUT "%s\n", $_;
        }
}

Note I didn't test with more than 10GB, maybe perl will overflow at some limit; to solve that, instead of counting bytes use Kbytes :

$maxsize=10*1024**2; # 10M of Kbytes
...
     $size +=( -s "$_")/1024;

EDIT: I noted that this first solution would not sort file by mtime, here is a more complete solution (similar to the bash script that has been posted by another person).

#!/usr/bin/perl
use File::stat;

$source="/somedir/";
$target="host:/remotedir";
$maxsize=10 * 1024**3; # 10GB  

open (RSOUT,"|rsync -av --files-from=- $source $target");
open (RSIN, "rsync -avn $source $target |");
while (<RSIN>)
{
    chomp;
    last if (/^$/);
    if (-f "$_")
    {
            my $fileattr;
            my $stat=stat($_);
            $fileattr->{name}=$_;
            $fileattr->{size}=$stat->size;
            $hash{sprintf ("%s %s\n", $stat->mtime, $_)}=$fileattr;
    }

}

foreach $key (reverse sort keys %hash)
{
    next if ( ($size + $hash{$key}->{size}) > $maxsize);
    $size += $hash{$key}->{size};
    print RSOUT $hash{$key}->{name}, "\n";
}
share|improve this answer

You can parse the sorted output of du. Assuming GNU utilities:

du -0ak | sort -z -k1n | awk -v 'RS=\0' -v 'ORS=\0' '
    (size += $1) > 10*1024*1024 {quit}
    {print substr($0, index(s, "\t")+1)}
' | xargs -0 cp -t destination

POSIXly, assuming that no file name contains a newline character:

du -ak | sort -k1n | awk '
    (size += $1) > 10*1024*1024 {quit}
    {print substr($0, index(s, "\t")+1)}
' | while IFS= read -r filename; do cp -- "$filename" /path/to/destination

Note that du traverses subdirectories. To avoid that, tell du which files you want to operate on. More generally, you can use find to filter files.

find . -type f ! -name excluded-file -exec du -ak {} + |
sort -k1n | awk '
    (size += $1) > 10*1024*1024 {quit}
    {print substr($0, index(s, "\t")+1)}
' | while IFS= read -r filename; do cp -- "$filename" /path/to/destination
share|improve this answer
    
is there a way to add rsync like functions ? this will be run more than once but this script will copy the files multiple times ? –  exussum Oct 23 '13 at 19:21
    
@user1281385 You can call rsync instead of cp. –  Gilles Oct 23 '13 at 22:52
    
the rysnc function would be to remove the old ones when run multiple times not to not transfer the file if already exists –  exussum Nov 28 '13 at 17:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.