I am looking for some assistance in trying to optimize a bro network log parsing script, here's the background:
I have a large amount of bro logs, but I'm only interested in querying IPs within my scope (multiple variable length subnets).
So I have a text file with regex patterns to match the IP ranges I'm looking for: scope.txt:
(scope.txt contains up to 20 more lines of other IP ranges in regex patterns) findInScope.sh:
#!bin/sh for file in /data/bro_logs/2016-11-26/conn.*.log.gz do echo "$file" touch /tmp/$file for nets in $(cat scope.txt) do echo "$nets" zcat $file | bro-cut -d | awk '$3 ~ '$nets' || $5 ~ '$nets'' >> /tmp/$file done sort /tmp/$file | uniq > ~/$file rm /tmp/$file done
As more background, each hour of original bro conn logs is about 100MBs, so my current script takes about 10-20 minute to parse through one hour of log data. One day of logs can take up to 3 hours.
I thought about a single awk statement with 40 or's but decided I don't want to do that because I want a separate scope.txt file in to use the same script for different scopes of IP ranges.
I also tried zcat on multiple conn.log files (i.e. zcat conn.*.log.gz) but the output file ended up being over 1GB, and I wanted to keep hourly logs intact.