GNU Parallel to process multiple files in parallel in Unix with bash

Question

I have a bash script I am using to iterate over a list of files and search for regex matches. Here is a snippet from my current code:

for file in $Files_To_Parse; do (cat $file) | grep -ioE "($Keys$Delimiters$Payload+$End_String)" | grep -v 'null' | grep -v '*' done

• $Files_To_Parse is a string of space separated file names

• $Keys, $Delimiters, $Payload$, and End_String are regex patterns.

This code currently works (slowly).

I would like to either have multiple files processed in parallel, or one file which regex matches are searched for in parallel; however, I am not sure how to use GNU's Parallel package to accomplish this.

Thanks for looking.

Answer 1

Be aware that your disks I/O might be limiting you:

doit() {
    (cat "$1") | grep -ioE "($Keys$Delimiters$Payload+$End_String)" | grep -v 'null' | grep -v '*'
}
export -f doit
parallel doit ::: $Files_To_Parse

Consider walking through the tutorial. Your command line will love you for it:

man parallel_tutorial