I need to iterate through the subdirectories of a directory and take two of the files, as arguments of an awk script. The script will compare the two files and generate other files.

I've this. But I need awk script to take as an argument files. ".*1.txt" and ".*2.txt"

for i in words/*/*1.txt words/*/*2.txt
awk -f corpus_vs_flexion.awk "$i"

Something like:

awk -f corpus_vs_flexion.awk .*1.txt .*2.txt
# Taking them from each subdirectory in words/* 

Directory words/
               subdirectory Peter/
               subdirectory Blas/
For each subdirectory loop: awk -f corpus_vs_flexion.awk whatever1.txt whatever2.txt
  • So what are the pairs? words/Peter/whatever1.txt and words/Blas/whatever2.txt? words/Peter/whatever1.txt and words/Peter/whatever2.txt? Are the pairs always in the same directory? – terdon Feb 29 '16 at 9:13
  • words/Peter/whatever1.txt and words/Peter/whatever2.txt are the pairs always in the same directory. For words/Peter/ awk -f corpus_vs_flexion.awk whatever1.txt whatever2.txt for words/Blas/ awk -f corpus_vs_flexion.awk whatever1.txt whatever2.txt – Firefly Feb 29 '16 at 9:17
  • OK, please edit your question and clarify that there are always pairs in the directories. Your multiple /same/path/whatever.txt are confusing. Also, will the files always end in 1.txt and 2.txt? Or can we have 3.txt and 4.txt or anything else? – terdon Feb 29 '16 at 9:23
  • If what you found solved your problem (and it looks like it does), add it as an answer and accept it if nobody comes up with another answer that you like better - it's perfectly acceptable to answer your own question on this site. Answers do not belong embedded in questions. – cas Feb 29 '16 at 9:47
  • @cas You're right!! Done! – Firefly Feb 29 '16 at 9:52

Ok, with file names occurring in matching pairs, you could use the following:

for f in words/*/*1.txt ; do awk -f corpus_vs_flexion.awk "$f" "${f%1.txt}2.txt" ; done

The "${f%1.txt}2.txt" phrase says "use the filename of "$f" but remove the ending 1.txt and add ending 2.txt instead".

  • Files to process in each subdirectory. End in "1.txt" and "2.txt". Example: "blabla1.txt" or "blublu2.txt", plus there may be more files with other different names. Then I have to use a regular expression to process – Firefly Feb 29 '16 at 8:45

I have been reading a bash book, and I found what I needed!


for ((i=0;i<=${#typ1_files[@]};i++)); do
   awk -f corpus_vs_flexion.awk "${typ1_files[i]}" "${typ2_files[i]}"
  • 1
    i'm unsure if your answer or @Ralph's answer is better. Both assume that there will always be a matching 2.txt for each 1.txt. Your answer has the potential to get out of sync (with elements from typ1_files not corresponding to matching elements from typ2_files - i.e. referring to unrelated 2.txt files in another directory) and messing up the entire run from the point that synchronisation failed. Ralph's answer will fail only for the specific file(s) where a matching 2.txt doesn't exist, a far less disastrous failure mode - i've changed my mind, Ralph's answer is better as it's safer. – cas Feb 29 '16 at 9:58
  • A simple safety net would be to add a line like [[ ${#typ1_files[@]} = ${#typ2_files[@]} ]] || exit so it will exit the program if the number of files in each array isn't the same. – terdon Feb 29 '16 at 10:00
  • yep. OTOH, that would also fail in the same way if there were 2.txt files without matching 1.txt files, so that the count remained equal. probably not an issue in real life with this set of files, but it's always worthwhile being paranoid and thinking about what could go wrong and programming defensively to avoid it. – cas Feb 29 '16 at 10:06

If you can be sure your file search is exhaustive and not matching other files, simplify by losing the for loop and just use a command substitution with find.

awk -f corpus_vs_flexion.awk $(find /path/to/your/dir -name "*.txt" -type f | tr '\n' ' ')
                                    ^^^^^^^^^^^^^^^^^ Put your dir here
  • I have to process, whatever 1.txt and whatever2.txt at the same time, in each loop. awk -f corpus_vs_flexion.awk whatever1.txt whatever2.txt for each subdirectory. – Firefly Feb 29 '16 at 8:58

I like Ralph's answer. This might also work (untested)

find. -name '*[12].txt' -print0 | xargs -0 -n 2 awk '...'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.