3

Problem

I cannot perform a recursive parallel zgrep while also omitting a string from the results.

I am zgrepping through about 640GB of compressed firewall logs and almost %30 of the lines have the string "Duplicate SYN" (which i am trying to omit)

Recursive Parallel Zgrep (without omitting a string) - success

I am able to successfully perform a parallel recurse grep like this

find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt > /tmp/bar.txt

Content of foo.txt:

10\.10\.0\.28
10\.10\.3\.41
10\.10\.0\.46
10\.10\.5\.47
10\.11\.0\.48
10\.10\.0\.49
10\.144\.41\.145
10\.122\.41\.241

Example Output

Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
May 31 02:58:46 ASAFW01 : %ASA-6-302014: Teardown TCP connection 461681145 for DMZ_EXP_INSIDE:10.122.41.241/7400 to inside:10.5.91.50/30378 duration 0:00:00 bytes 0 Failover primary closed
May 31 02:58:47 ASAFW01 : %ASA-6-302013: Built outbound TCP connection 1962428108 for DMZ_EXP_INSIDE:10.122.41.241/7400 (10.122.41.241/7400) to inside:10.11.0.48/33990 (10.11.0.48/33990)
May 31 02:58:47 ASAFW01 : %ASA-6-302014: Teardown TCP connection 1962428108 for DMZ_EXP_INSIDE:10.122.41.241/7400 to inside:10.11.0.48/33990 duration 0:00:00 bytes 3188 TCP Reset-O from DMZ_EXP_INSIDE
May 31 02:58:49 ASAFW01 : %ASA-6-106015: Deny TCP (no connection) from 10.11.0.48/35976 to 10.122.41.241/7400 flags RST  on interface inside
May 31 02:58:49 ASAFW01 : %ASA-6-106015: Deny TCP (no connection) from 10.11.0.48/35976 to 10.122.41.241/7400 flags RST  on interface inside

Output I want to omit

Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number

Ideas

  • Modify foo.txt to use a regular expression that omits the word "Duplicate" (not sure how to do this)
  • Recursively remove all lines that contain the word "Duplicate" in all 640GB of gzipped log files (using sed?)

Recursive Parallel Zgrep (with omitting a string) - failure

However when I attempt to exclude something from the results I get an error.

Command that causes an error:

find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt -v Duplicate > /tmp/bar.txt

gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
|improve this question
  • can you create two very small sample files (no need compressed, plain needed for testing) - and show what you expect from the grep command? I don't think you can specify both a positive search string and a negative one in same command.. you'll have to pipe output of one to the other, or may be possible with PCRE (but expensive, piping may still be faster) – Sundeep Jul 10 '19 at 13:45
  • 1
    try grep -f foo.txt ip.txt | grep -v Duplicate .. also, you can remove all the backslashes from foo.txt and use grep -Ff foo.txt ip.txt | grep -v Duplicate.. I would further suggest to use grep -wFf foo.txt ip.txt | grep -v Duplicate to avoid 10.10.0.28 matching something like 10.10.0.282 or 210.10.0.28 – Sundeep Jul 10 '19 at 15:24
  • I do not want to pipe it in to a "-v" as i don't have space to handle the temp file that will balloon as a result of including the "duplicate" lines – brakertech Jul 10 '19 at 16:11
  • I'd modify foo.txt to have (?!.*Duplicate) starting each line and have zgrep use -P so that the negative lookahead works. Also, Sundeep's got a good point about substrings... might use \b at the start and end of each IP to ensure there's a word boundary. – stevesliva Jul 10 '19 at 16:25
  • @stevesliva when i used -P i got the error message "grep: the -P option only supports a single pattern" – brakertech Jul 10 '19 at 16:30
1

The easiest way to write this additional condition is simply to pipe your output through another invocation of grep before writing to the file:

find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt | grep -v Duplicate > /tmp/bar.txt

The performance of doing this in a separate single-threaded process will depend on how many records are being returned by your search. If the number of lines matching your initial search is only a small subset of your 640GB of logs, it shouldn't be an issue.

Since the underlying question is also partly about optimizing this search overall, I'll note that there are two big additional enhancements you can make:

  1. Use fixed strings rather than regular expressions for your grep search terms -- remove the backslashes from your pattern file and add the -F switch to zgrep. (The -w switch to only match whole words would also likely be a good idea.) Searching for literal strings is far faster than searching for regular expressions.
  2. If your system's locale uses UTF-8 but your data set is only ASCII, set LC_ALL=C in the command's environment. The search will go much faster working within the 128-character ASCII set than it will in the ~1000-times-larger UTF-8 character set.
|improve this answer
  • Using the output from xargs -P is dangerous due to line mixing: mywiki.wooledge.org/… – Ole Tange Jul 14 '19 at 5:28
  • @OleTange: Your comment belongs on the question, because the OP is already using the output from xargs -P.  Dan is just building on the command in the question. – G-Man Says 'Reinstate Monica' Nov 23 '19 at 2:27
1

If you have GNU Parallel you can do:

find /var/logs/syslog -name \* -print0 |
  parallel --lb -0 'zgrep -f foo.txt {} | grep -v Duplicate' > /tmp/bar.txt

Opposite xargs -P using the output from GNU Parallel is guaranteed not to mix lines (as per mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P).

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.