3

I have a directory with a bunch of files with names like a04x.txt, each with a corresponding b04y.txt file. I need to be able to run some commands on each pair of files and produce an additional file c04z.txt for each pair.

The actual numbers on the files are rather large and pretty sparse, so simply iterating over all numbers from 1 to 99 or something like that won't work.

Currently I use the following to handle the task, but seems like a common enough task that there ought to be a shorter/better way to do it:

for num in ./a*x.txt
do
  num="${num##*/a}"
  num="${num%x.txt}"

  my_command a${num}x.txt b${num}y.txt c${num}z.txt
done

Ideally I would also like to be warned when there are a${num}x.txt or b${num}y.txt files that don't have a matching file with the same number. I'd also like an easy way to be able to just pipe the sets of files to xargs or parallel so I can have it process multiple sets of files simultaneously.

Is there a better way to do this?

  • Gather all a*x.txt and b*y.txt in an array, extract the numbers from the filenames, select only the duplicate numbers with uniq -d then pipe the result to xargs or parallel, e.g. myfiles=(a*x.txt b*y.txt); printf %s\\n "${myfiles[@]//[^0-9]/}" | sort | uniq -d | parallel cmd a{}x.txt b{}y.txt c{}z.txt – don_crissti Mar 15 '16 at 18:22
  • @don_crissti, you may as well post that as an answer. – Wildcard Mar 15 '16 at 22:58
  • @Wildcard - I know that thread over on the meta :) - however, the question here is not clear as far as I am concerned. OP says "I would also like to be warned..." but doesn't explain what should the script do if there are unpaired files ? Should it warn and skip or should it warn but still run my_command (there's a possibility that my_command is smart and still runs fine even if some files don't exist - we don't know that...) ? So yeah, there are a couple of things to clarify here. But anyway, if anyone wants to turn my comments into answers, they are welcome to do so, anytime. – don_crissti Mar 15 '16 at 23:22
2
  1. One approach would be to do

    for afile in a*x.txt
    do
        bfile=${afile/a/b}; bfile=${bfile/x.txt/y.txt}
        cfile=${afile/a/c}; cfile=${cfile/x.txt/z.txt}
    
        my_command "$afile" "$bfile" "$cfile"
    done
    

    although I guess that isn’t a big improvement, and it could fail in a pathological case like a filename of afoox.txtbarx.txt.  Also, note that this is specifically a bash feature; it might not work in other POSIX-compliant shells (unlike ## and %, which are specified by POSIX).

  2. It’s a simple matter to say

        if [ -f "$bfile" ]
        then
            my_command "$afile" "$bfile" "$cfile"
        else
            echo Error
        fi
    

    to catch a file outliers (e.g., a17x.txt with no corresponding b17y.txt).

  3. If you put

    for afile               # with no list, defaults to "$@"; i.e., the script’s arguments
    do
        bfile=${afile/a/b}; bfile=${bfile/x.txt/y.txt}
        cfile=${afile/a/c}; cfile=${cfile/x.txt/z.txt}
    
        if [ -f "$bfile" ]
        then
            my_command "$afile" "$bfile" "$cfile"
        else
            echo Error
        fi
    done
    

    into a script, then you can run that script with a list of anumx.txt filenames as arguments, and it will process them.  You can then run that script through xargs or parallel.

  4. Checking for b file outliers (e.g., b42y.txt with no corresponding a42x.txt) as part of the above process is not straightforward, but it is easy to do a separate loop:

    for bfile in b*y.txt
    do
        afile=${bfile/b/a}; afile=${afile/y.txt/x.txt}
        if [ ! -f "$afile" ]
        then
            echo Error
        fi
    done
    
  • If you're using globs why not just specify what you want? for afile in a[0-9][0-9]x.txt – Wildcard Mar 15 '16 at 23:38
  • 1
    There is some ambiguity about what the OP wants.  The first version of the question says "The actual numbers on the files are sparse, may be quite large, and in a few cases are not just numeric", so it would seem to be inappropriate to assume that the filenames are all a, followed by a two-digit number, followed by x.txt.  P.S. Note also that even the current version of the question talks about using a*x.txt; if you want to criticize the OP's approach, perhaps you should do it in a comment on the question. – G-Man Mar 16 '16 at 1:56
2

GNU parallel has a way to do this, and runs the commands in parallel as a bonus:

$ parallel my_command {} \
                      {= s/a([0-9]+)x.txt/b\1y.txt/ =} \
                      {= s/a([0-9]+)x.txt/c\1z.txt/ =} \
           ::: a*x.txt

Those substitutions are perl code. The line breaks are just for readability -- this is a one-liner.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.