Problem
I cannot perform a recursive parallel zgrep while also omitting a string from the results.
I am zgrepping through about 640GB of compressed firewall logs and almost %30 of the lines have the string "Duplicate SYN" (which i am trying to omit)
Recursive Parallel Zgrep (without omitting a string) - success
I am able to successfully perform a parallel recurse grep like this
find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt > /tmp/bar.txt
Content of foo.txt:
10\.10\.0\.28
10\.10\.3\.41
10\.10\.0\.46
10\.10\.5\.47
10\.11\.0\.48
10\.10\.0\.49
10\.144\.41\.145
10\.122\.41\.241
Example Output
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
May 31 02:58:46 ASAFW01 : %ASA-6-302014: Teardown TCP connection 461681145 for DMZ_EXP_INSIDE:10.122.41.241/7400 to inside:10.5.91.50/30378 duration 0:00:00 bytes 0 Failover primary closed
May 31 02:58:47 ASAFW01 : %ASA-6-302013: Built outbound TCP connection 1962428108 for DMZ_EXP_INSIDE:10.122.41.241/7400 (10.122.41.241/7400) to inside:10.11.0.48/33990 (10.11.0.48/33990)
May 31 02:58:47 ASAFW01 : %ASA-6-302014: Teardown TCP connection 1962428108 for DMZ_EXP_INSIDE:10.122.41.241/7400 to inside:10.11.0.48/33990 duration 0:00:00 bytes 3188 TCP Reset-O from DMZ_EXP_INSIDE
May 31 02:58:49 ASAFW01 : %ASA-6-106015: Deny TCP (no connection) from 10.11.0.48/35976 to 10.122.41.241/7400 flags RST on interface inside
May 31 02:58:49 ASAFW01 : %ASA-6-106015: Deny TCP (no connection) from 10.11.0.48/35976 to 10.122.41.241/7400 flags RST on interface inside
Output I want to omit
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Apr 18 01:39:33 ASAFW01 : %ASA-4-419002: Duplicate TCP SYN from inside:10.10.0.28/61763 to inside:10.122.41.241/8443 with different initial sequence number
Ideas
- Modify foo.txt to use a regular expression that omits the word "Duplicate" (not sure how to do this)
- Recursively remove all lines that contain the word "Duplicate" in all 640GB of gzipped log files (using sed?)
Recursive Parallel Zgrep (with omitting a string) - failure
However when I attempt to exclude something from the results I get an error.
Command that causes an error:
find /var/logs/syslog -name \* -print0 | xargs -0 -n 1 -P 36 zgrep -f foo.txt -v Duplicate > /tmp/bar.txt
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
gzip: Duplicate.gz: No such file or directory
grep -f foo.txt ip.txt | grep -v Duplicate
.. also, you can remove all the backslashes from foo.txt and usegrep -Ff foo.txt ip.txt | grep -v Duplicate
.. I would further suggest to usegrep -wFf foo.txt ip.txt | grep -v Duplicate
to avoid10.10.0.28
matching something like10.10.0.282
or210.10.0.28
– Sundeep Jul 10 '19 at 15:24(?!.*Duplicate)
starting each line and have zgrep use-P
so that the negative lookahead works. Also, Sundeep's got a good point about substrings... might use\b
at the start and end of each IP to ensure there's a word boundary. – stevesliva Jul 10 '19 at 16:25