What appropriate regex for recognizing a variable length file name?

Question

I have a bash script that classifies and retrieves files from a remote server. I have trouble with the classification step, which is based on the file names.

I am correctly able to identify the different families of files that are defined by the start of the filenames. Filenames can be:

ala-olo_ulu-1602915797.txt
ala-olo_ulu-1602915799.txt
ili-olo-1602915897.txt
ili-olo-1602915997.txt
ili-olo-pip-1602925797.txt
ili-olo-pip-1602935797.txt

In this example, I have 3 families: ala-olo_ulu ili-olo ili-olo-pip (pure examples :))

Each family is treated in one iteration of a loop. In such an iteration, I have the family name available in an variable BASE_NAME (for instance ili-olo).

My trouble is the taring step, before rsync'ing the files to local. I am managing it with the following ssh command.

      ssh root@"${RMT_IP}" '
          for FILE in "'${BASE_NAME}'*'${FILE_EXTENSION}'"; do
            tar -rf "'${BASE_NAME}'.tar" ${FILE} --remove-files
          done' < /dev/null

With this script, unfortunately, if ili-olo is managed before ili-olo-pip, then, the archive will contain both families (they both share the same start). And when ili-olo-pip will be then managed, they won't be any file anymore, and the tar command ends in error. (which is how I could spot the trouble).

I guess, I should rather use regex to specify that the variable part of the file name is the digit part. Please, how can I change the for loop definition so that the families starting with the same string do not get into the same tar?

for FILE in "'${BASE_NAME}'*'${FILE_EXTENSION}'"; do

?

The digit part always has the same number of digits (it is a timestamp, with second precision), for instance 1602915797

I thank you for your help. Have a good day, Bests, Pierre

Answer 1

It's easier if you can use zsh as both the local and remote shell:

ssh root@$RMT_IP zsh << EOF
  set -o extendedglob # for (#c10)

  for file in ${(qq)BASE_NAME}-[0-9](#c10).${(qq)FILE_EXTENSION}(N); do
    tar -rf ${(qq)BASE_NAME}.tar \$file --remove-files
  done
EOF

[0-9](#c10) matches a sequence of 10 decimal digits. See also [0-9]## same as [0-9](#c1,) for one or more digits or <100000-9999999999> (which doesn't require extendedglob) for sequences of decimal digits making up numbers in that range.

sshd on the server runs the login shell of the user to interpret the code passed as argument. Since we don't know what it is (often for root, that's just sh), we just make that code zsh, to start a zsh shell and pass the zsh code on stdin.

Using a here-document like that makes it easier to construct the shell code to be interpreted by the remote shell there. As the EOF is not quoted, the local shell will perform expansions locally.

It's important to keep track of which expansions are meant to be done locally and which are meant to be done by the remote shell.

Above ${(qq)BASE_NAME} is expanded by the local shell, we use the (qq) parameter expansion flag to quote the result with single quotes, so that the remote shell takes it as a literal string.

$file has to be expanded by the remote shell, so we prefix it with \ so that a literal $file be passed to the remote shell.

If zsh is not available on the remote machine, but bash is, you could do (still using zsh locally):

ssh root@$RMT_IP bash --norc << EOF
  shopt -s extglob nullglob # for +(...)
  export LC_ALL=C

  for file in ${(qq)BASE_NAME}-+([0-9]).${(qq)FILE_EXTENSION}; do
    tar -rf ${(qq)BASE_NAME}.tar "\$file" --remove-files
  done
EOF

bash doesn't have the equivalent of zsh's x(#c10) glob operator, but with extglob, it supports a subset of the ksh ones (not {10}(x) though unfortunately here), including +(x) which matches one or more x. So that +([0-9]) will match one or more digits instead of just 10.

To match 10 digits, you could still do [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].