List transient files by numerical value having number greater than a given constantly varying filename

Question

I've a directory having constantly changing filenames (by numerical value) listed as:

    -rw-rw----. 1 root root 10493952 May  7 10:39 A0000000.LOG
    -rw-rw----. 1 root root 10493952 May  7 08:38 A0000001.LOG
    -rw-rw----. 1 root root 10493952 May  7 08:38 A0000002.LOG
    ...
    ...
    -rw-rw----. 1 root root 10493952 May  7 08:38 A0000582.LOG
    ...
    and so on...

Now, at any given moment, my app throws a filename from these listed files. I've to discard that file and any other file having greater numerical value than that file from rsyncing to remotehost

Say application throws file A0000096.LOG. And I if I see A0000097.LOG getting created as well, I would::

rsync A* --exclude A0000096.LOG --exclude A0000097.LOG user@remoteHost:/somedir/

• Points to note:

  • App throws different file name to rsync cron job running every hour. In this example, filename is A0000096.LOG

  • Files having higher numerical value than App's output filename (A0000096.LOGin above example) can have same create date/timestamp as App's output filename

  • By the time rsync starts, App's output filename can have more recent altered date/timestamp than files having higher numerical value.

  • There may be a possibility that higher numerical value files not getting created or may get created more than just 1 (could be 10 to 20)

Question: With these restrictions, I'm not able to find how can I provide all higher numerical value files to my rsync bash script so that I can exclude them.

Any help is appreciated.

Answer 1

If I understand the question correctly, you want to rsync all the files in the range A0000000 through A0000095.  Well, then, don’t say A*; use a list of positive wildcards (a.k.a. globs or filename expansion patterns) that generate the file names you want, rather than identifying the ones you want to exclude.  Do it by decomposing the range into subranges:

         Subrange                         Wildcard
    A0000000-A0000089                 A00000[0-8][0-9]
    A0000090-A0000095                 A000009[0-5]

So you would say

rsync A00000[0-8][0-9] A000009[0-5] user@remoteHost:/somedir/

In case you have trouble generalizing this approach from one example, consider 97169.

         Subrange                           Wildcard
    A0000000-A0089999                 A00[0-8][0-9][0-9][0-9][0-9]
    A0090000-A0096999                 A009[0-6][0-9][0-9][0-9]
    A0097000-A0097099                 A00970[0-9][0-9]
    A0097100-A0097159                 A00971[0-5][0-9]
    A0097160-A0097168                 A009716[0-8]

It is conceivable that some of these wildcards will not match any existing files.  In that case, do

shopt -s nullglob

to tell the shell to go ahead and run the rsync command with the failing wildcards ignored (i.e., expanded to null).

Answer 2

Since the numbers are padded with zeros to the same width, the numerical order is identical to the lexicographic order. Therefore your problem is equivalent to removing the files starting with a given file in the lexical order.

You can do this by building a string containing the file names separated by newlines, and using string substitution to remove the end of the string, then relying on unquoted expansion to turn the truncated string back into a list. I assume that there are no newlines in file names and that your shell is bash or ksh (with plain sh, you need to use positional parameters instead of a named array).

nl=$'\n'                              # newline, we use it as a separator
cut_from=A0000096.LOG
log_files=(A???????.LOG)
set -f; IFS="$nl"                     # disable wildcard expansion and set the word separator to newline only
log_files="$nl${log_files[*]}$nl"     # turn the array into a string with newlines separating elements
log_files=(${log_files%"$nl$cut_from$nl"})    # remove elements from $cut_from onwards and split the string into an array
unset IFS; set +f
rsnyc -a "${log_files[@]}" … elsewhere:/some/dir

Answer 3

Answering my own question:

Method 1

  CUT_LOG=A0000096.LOG        #Assuming app throws a file - A0000096.LOG and I've to exclude this and any other files having higher numerical value than this file: A0000096.LOG. e.g. A0000097.LOG etc.

   LISTLOG=`ls -1 /Source_Dir/A*.LOG | sed "/$CUT_LOG/,$ d"`       # Instantiate a variable - LISTLOG, that will hold files list excluding A0000096.LOG and any other files having higher numerical value than this A0000096.LOG


      rsync -a `echo "${LISTLOG[@]}"` user@remoteHost:/somedir/    # Transfers the files excluding A0000096.LOG and excluding any other files with higher numerical values than A0000096.LOG

Method 2 [using a loop (Not recommended)

        for line in `echo "${LISTLOG[@]}"`; do                     # Recursively loop through the array and rsync each entry
                rsync -a $line user@remoteHost:/somedir/
        done

Answer 4

Tangenting off Gilles' observation that your filenames sort lexically, you could use zsh to create a filter file for rsync:

cut_log=A0000096.LOG zsh -c 'logfilter() { ! [[ "$REPLY" < "$cut_log" ]] }; print -l A*.LOG(+logfilter)' > .exclude_file
rsync A* --exclude-from=.exclude_file user@remoteHost:/somedir/
rm .exclude_file

Of course, you could do it all natively in zsh, but I'm demonstrating how to extend your bash script with a helpful zsh feature. This feature is called a "Glob Qualifier" and is described in the eString and +cmd section. It asks for the expansion of A*.LOG filtered by passing each candidate through the given function. The function's parameter is named REPLY, so we compare that lexically to the value of $cut_log to determine inclusion or exclusion in the resulting list.

The above assumes that the code is executing in the directory that contains the log files, both so that rsync finds them and so that zsh finds them.

This allows an easy dry-run by inspecting the .exclude_file before executing the rsync command.

If you find that there are too many files to include on the command-line (such that the command fails because there are too many arguments), but there are few enough exclude files, you could invert the logic and tell rsync to --include-from a file, then switch the logfilter logic to remove the inversion (!).