2

Trying to complete a script with the following steps: (1) select 1gig of most recently changed files across a number of subdirectories (2) preferably uses rsync to copy the files to local directory -- I prefer rsync than cp as I can use features from rsync to skip existing files etc.

for step 1, the following works and it gives me the newest files limited to 1gig

ls -lstrkR /volume1/cctv/* | grep \.mp4$ | awk ' 
  (size += $1) > 1*1024*1024 {exit}
  #{print "size=" size "\t" $1 "\t" $6 "\t" $7 " " $8 " "$9 "\t" $10}
  {print $10}
'

The output from the above looks like this: file1.mp4 file2.mp4 etc.

I do not have the absolute path of each file and the files above are from several subdirectories of /volume1/cctv (as you can see ls -R)

I need to either: (a) take the output from the above and pipe to rsync, OR (b) execute cp on the files (but then can I work from this list without the absolute paths?)

  • don't throw away information (i.e. the full path to the file) that you'll need to use later. in other words, apart from it being a bad idea in general to parse ls, don't use ls -R. use something like find /volume1/cctv/* -type f -iname '*.mp4' -exec stat -c '%Y\t%s\t%n\n' {} + | awk '...' instead. read man stat for details on the -c format string. and note that this will fail if any filenames have linefeeds in them. – cas Nov 13 '15 at 16:46
  • the stat command outputs mod-time in seconds since epoch, file size in bytes, and filename for each matching fle. separated by tab characters, not spaces. – cas Nov 13 '15 at 16:48
  • thanks, but.. I am trying find -type f -iname '*.mp4' -exec stat -c '%Y\t%s\t%n\n' {} + to see what comes out from that first and I get an error find: -exec CMD must end by ';' .. the problem with the above though is that I lose the sort which I get from ls. If I can get find to work, how would I sort by date then? – csolad Nov 13 '15 at 16:57
  • change + to \;. what OS are you running and what version of find? you don't need the sort, you can do that in awk, that's why the stat includes the file size as the second field. or put something like sort -n -k2,2 in the pipeline between find and awk. – cas Nov 13 '15 at 17:00
  • if your find doesn't understand -exec + then your stat probably doesn't understand -c. check the man page to see if there is an equivalent option for your version of stat. – cas Nov 13 '15 at 17:03
2

This perl script should do what you want: Given a NUL-separated list of filenames (e.g. from find -print0), output a list of the most recently modified filenames, as long as the total size of those files doesn't exceed 1GB (default). You can specify the number of gigs for the maximum size on the command line - this can be any valid number, integer or floating point.

The NUL separator means that this will work with any filenames, even if they contain spaces or newlines.

$ cat select-newest-one-gig.pl
#! /usr/bin/perl -0

use strict;

my $gigs = shift || 1;

my $maxsize = $gigs * 1024 * 1024 * 1024 ;  # 1GB
my $total = 0;

# a hash to contain the list of input filenames and their modtimes
my %filemtimes=();

# hash to contain the list of input filenames and their sizes
my %filesizes=();

# a hash to contain a list of filenames to output.
# use a hash for this so we don't need to write a `uniq` function.
my %outfiles=();

while (<>) {
   chomp;

   # 7th field of stat() is size in bytes.
   # 9th field of stat() is modime in secs since epoch

   my ($size,$mtime) = (stat($_))[7,9];
   $filesizes{$_} = $size;
   $filemtimes{$_} = $mtime;
}

# iterate through the %filemtimes hash in order of reverse mtime
foreach (reverse sort { $filemtimes{$b} <=> $filemtimes{$a} } keys %filemtimes) {
   my $size = $filesizes{$_};

   # add it to our list of filenames to print if it won't exceed $maxsize
   if (($size + $total) <= $maxsize) {
       $total += $size;
       $outfiles{$_}++;
   }
}

# now iterate through the %filesizes hash in order of reverse size
# just in case we can sequeeze in a few more files.
foreach (reverse sort { $filesizes{$b} <=> $filesizes{$a} } keys %filesizes) {
   my $size = $filesizes{$_};
   if (($size + $total) < $maxsize) {
       $total += $size;
       $outfiles{$_}++;
   }
}

# now print our list of files.  choose one of the following, for
# newline separated filenames or NUL-separated.   
#print join("\n", sort keys %outfiles), "\n";
print join("\000", sort keys %outfiles), "\000";

Save that as select-newest-one-gig.pl and make it executable with chmod +x.

Run it like this (e.g. for a max total file size of 10GB):

find /volume1/cctv/ -type f -iname '*.mp4' -print0 | ./select-newest-one-gig.pl 10

This perl script could fairly easily be modified to take one or more filename extensions (e.g. .mp4) as args and then run find itself using the system() function call and iterate over that rather than while (<>). It's probably simpler to just pipe find's output into it - why re-invent the wheel?

The following perl script will list (or delete if you uncomment the last line) files that exist in the rsync target directory that weren't listed on stdin. It assumes NUL-separated input, so is safe even with filenames that contain newlines.

$ cat unlink-others.pl
#! /usr/bin/perl -0

use strict;

my @files=();

# first arg is target dir, with default
my $targetdir = shift || '/path/to/rsync/target/dir/';

while (<>) {
    chomp;
    s/^.*\///;  # strip path
    push @files, quotemeta($_)
}
my $regexp=join("|",@files);

opendir(my $dh, $targetdir) || die "can't opendir $targetdir: $!\n";
my @delete = grep { ! /^($regexp)$/o && -f "$targetdir/$_" } readdir($dh);
closedir $dh;

print join(", ",@delete),"\n";
# uncomment next line if you're sure it will only delete what you want
# unlink @delete

Use it like this:

find /volume1/cctv/ -type f -iname '*.mp4' -print0 | \
    ./select-newest-one-gig.pl 10 > /tmp/files.list

rsync --from0 --files-from /tmp/files.list ... /path/to/rsync/target/dir/

./unlink-others.pl /path/to/rsync/target/dir/ < /tmp/files.list
  • if your find doesn't support -print0 then remove it from the find command-line and also remove the -0 from the #! line of the perl script. This will lose the ability to safely handle filenames that have newlines in them. – cas Nov 14 '15 at 10:10
  • cas thank you for the above. Truth is I've been trying since last Saturday to piece together the above script into the overall solution to achieve the result I need and I keep running into issues. I can take the output of the above, create a text file. I can then push that text file to rsync --files-from=filelist.txt ... the problem now is to delete from the target directory the files which are NOT in the filelist.txt -- I am thinking to iterate the target directory and delete the file if it does not exist in the filelist.txt - are you able to help me with that please? – csolad Nov 20 '15 at 10:27
  • why not just use rsync's --delete option? – cas Nov 20 '15 at 12:14
  • I tried, its not working. here is what I have done rsync --times --recursive --delete --progress --files-from=./out5.txt / /volume1/gDrive-CCTV/ it doesnt delete them. When I searched for --delete not working, there are a lot of hits saying that --delete doesn't work. arghh – csolad Nov 20 '15 at 14:52
  • full output: admin@SAM:~>rsync --times --recursive --delete --progress --files-from=./out5.txt / /volume1/gDrive-CCTV/ sending incremental file list rsync: link_stat "/volume1/s/1/20151118PM/Garage20151118-234246-1447890166.mp4" failed: No such file or directory (2) sent 2628 bytes received 24 bytes 5304.00 bytes/sec total size is 6433064699 speedup is 2425740.84 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1240) [sender=3.0.9] – csolad Nov 20 '15 at 14:55
0
cd /volume/cctv
echo 0 >/tmp/sztally &&
find .// -name '*.[mM][pP]4' -type f -exec sh -fc '
     _cp(){ shift; echo cp "$@$0"; :; }
     read sz </tmp/sztally; IFS=/ g=$((1024*1024)); unset _f
     for   f in   $(ls -dkst "$@")
     do    case   $f  in
           ("")   f=${2+./${_f%[!0-9. ]*}} _f=${_f##*[pP]4?}
                  [ 0 -ne "$((g>(sz+${_f%??})))" ] &&
                  set "$f$@" && sz=$((sz+${_f%??})) _f=;;
           (*)    [ -z ${_f:+:} ] && set "" ${_f+"$@"}
                  _f=${_f:+$_f/}$f
           esac||  ! _cp "$@" || exit 255
     done; _cp "$@"; echo "$sz" >/tmp/sztally
'   "/destination/path" {} +

That works for me. I tested it in my own media directory, and it consistently aggregated only the newest 1gb worth of .mp4 files into a cp operation. I think the ls option you were looking for was -d, which will retain full pathnames for all of ls's arguments. Here find seeks out all of the .mp4 files it can put together, and ls sorts its selections by modification time. The shell splits ls's output on pathname delimiters - the / - and so there is no issue with special characters in filenames, because special characters are never considered at all.

Strictly speaking, the -s option for ls doesn't report a file's size, but rather it reports a file's used space. These two concepts can differ, but in the case of compressed video files the likelihood that they do is surpassing small. This doesn't actually do a copy as written - it only echo's the cp operation. If you test it and find it viable, remove the echo from the _cp() function.

This depends on a POSIX ls, find, cp, and sh.

  • thanks, I haven't run it yet but I cannot see an rm above. Remember I want to delete from the target directory files which are NOT in the most recent 1gb. So, step (1): select most recent 1gb and copy to target, step (2): delete from target. I prefer not to over-write existing files in target because the target-dir is synced with googledrive. I don't want to re-upload every day 1gig of data if only say 200MB has changed (the actual data limit I want to stay in is 100gig not 1gig). That is why I wanted to use rsync but I dont mind as long as cp ignores existing. Does this make sense? – csolad Nov 23 '15 at 9:13
  • @csolad - ok, so use rsync rather than cp. or pax. or ln. feel free to adapt it in any way you might require. as i said, as written it only echoes a filelist to the screen. – mikeserv Nov 23 '15 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.