Parallel Recursive Grep While Omitting String

Question

Problem

I cannot perform a recursive parallel zgrep while also omitting a string from the results.

I am zgrepping through about 640GB of compressed firewall logs and almost %30 of the lines have the string "Duplicate SYN" (which i am trying to omit)

Recursive Parallel Zgrep (without omitting a string) - success

I am able to successfully perform a parallel recurse grep like this

find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt > /tmp/bar.txt

Content of foo.txt:

10\.10\.0\.28
10\.10\.3\.41
10\.10\.0\.46
10\.10\.5\.47
10\.11\.0\.48
10\.10\.0\.49
10\.144\.41\.145
10\.122\.41\.241

Example Output

Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
May 31 02:58:46 ASAFW01 : %ASA-6-302014: Teardown TCP connection 461681145 for DMZ_EXP_INSIDE:10.122.41.241/7400 to inside:10.5.91.50/30378 duration 0:00:00 bytes 0 Failover primary closed
May 31 02:58:47 ASAFW01 : %ASA-6-302013: Built outbound TCP connection 1962428108 for DMZ_EXP_INSIDE:10.122.41.241/7400 (10.122.41.241/7400) to inside:10.11.0.48/33990 (10.11.0.48/33990)
May 31 02:58:47 ASAFW01 : %ASA-6-302014: Teardown TCP connection 1962428108 for DMZ_EXP_INSIDE:10.122.41.241/7400 to inside:10.11.0.48/33990 duration 0:00:00 bytes 3188 TCP Reset-O from DMZ_EXP_INSIDE
May 31 02:58:49 ASAFW01 : %ASA-6-106015: Deny TCP (no connection) from 10.11.0.48/35976 to 10.122.41.241/7400 flags RST  on interface inside
May 31 02:58:49 ASAFW01 : %ASA-6-106015: Deny TCP (no connection) from 10.11.0.48/35976 to 10.122.41.241/7400 flags RST  on interface inside

Output I want to omit

Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number

Ideas

• Modify foo.txt to use a regular expression that omits the word "Duplicate" (not sure how to do this)
• Recursively remove all lines that contain the word "Duplicate" in all 640GB of gzipped log files (using sed?)

Recursive Parallel Zgrep (with omitting a string) - failure

However when I attempt to exclude something from the results I get an error.

Command that causes an error:

find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt -v Duplicate > /tmp/bar.txt

gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory

Answer 1

The easiest way to write this additional condition is simply to pipe your output through another invocation of grep before writing to the file:

find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt | grep -v Duplicate > /tmp/bar.txt

The performance of doing this in a separate single-threaded process will depend on how many records are being returned by your search. If the number of lines matching your initial search is only a small subset of your 640GB of logs, it shouldn't be an issue.

Since the underlying question is also partly about optimizing this search overall, I'll note that there are two big additional enhancements you can make:

1. Use fixed strings rather than regular expressions for your grep search terms -- remove the backslashes from your pattern file and add the -F switch to zgrep. (The -w switch to only match whole words would also likely be a good idea.) Searching for literal strings is far faster than searching for regular expressions.
2. If your system's locale uses UTF-8 but your data set is only ASCII, set LC_ALL=C in the command's environment. The search will go much faster working within the 128-character ASCII set than it will in the ~1000-times-larger UTF-8 character set.

Answer 2

If you have GNU Parallel you can do:

find /var/logs/syslog -name \* -print0 |
  parallel --lb -0 'zgrep -f foo.txt {} | grep -v Duplicate' > /tmp/bar.txt

Opposite xargs -P using the output from GNU Parallel is guaranteed not to mix lines (as per mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P).