1

I am examining Bro logs and would like to find a way to use AWK to grab IPs from a tracker link and search the Bro logs for those IPs automatically, printing the results of each match into [custom-name].txt file.

MORE INFO:

I am new to awk but trying to piece things together. I am not beholden to awk if there is a better way to do this. Efficient would be good.

Right now I am manually using an Excel function to concatenate the column of bad IPs (which are updated regularly from the link) into a super long awk OR string:

cat http* | awk '/123\.456\.789\.012/ || /2nd IP/ || /3rd IP/' | sort -n > file.txt

Currently, the http.log has the normal Bro headers... TS UID ID.orig_h ID.orig_p etc.

In layman's terms, what I'm trying to do is:

If the IP on line 1 of link is found in http.log, print results to line_1_ip.txt, If the IP on line 2 of link is found in http.log, print results to line_2_ip.txt, et al.

A snippet of the IPs would look like:

#############################################################################################
# abuse.ch ZeuS IP blocklist "BadIPs" (excluding hijacked sites and free hosting providers) #
#                                                                                           #
# For questions please refer to https://zeustracker.abuse.ch/blocklist.php                  #
#############################################################################################

103.230.84.239
103.241.0.100
103.26.128.84
103.4.52.150
103.7.59.135
104.218.48.99
104.219.232.12
104.236.55.19
107.150.58.84
108.61.193.211

It is just a text file.

When it comes to actually referencing a remote webpage to pull IPs from, I am honestly not sure where to even start. But if I can get it to pull from a local text file, that will give me a starting point from which to work.

  • 2
    What is the format of those log files? What output do you expect exactly? Show us what you've got so far too so we can help you make it work. – Mat May 2 '15 at 7:52
  • Is it need to be with awk? The same can be done with sample for loop and grep – Romeo Ninov May 2 '15 at 8:06
  • 1
    Please edit your question and i) include a sample of your tracker file (don't expect us to open random links in order to hep you. You can use the formatting tools to format it as code; ii) include a sample of these "Bro logs"; iii) show us the output you would like to have from those files; iv) show us what you've tried so far so we don't give you the same solutions and so we know you've actually put some effort into it and don't expect other people ti just do your work for you. If you edit to add all these, your question can be reopened. – terdon May 2 '15 at 9:03
  • @terdon, I have updated the post. Please advise if you still need more information. – user112802 May 3 '15 at 8:42
  • That's much better, thanks. Also, it wasn't an issue with the specific link you posted, we just like to have all information needed to answer the question here. Could you also show us an example of your BRO logs? What are the http* files you are cating? Are those the logs? – terdon May 4 '15 at 13:07
0

I'm not entirely sure on what you want to parse since I haven't seen the input format but, if you just want to read a list of IPs and print to the corresponding output file, you could try this:

awk 'NR==FNR{a[$1]=NR; next} {
    for(ip in a){ 
        if($0~ip){ 
            print >> "line_"a[ip]"_ip.txt" 
        } 
    }
}}' <(wget -O - "https://zeustracker.abuse.ch/blocklist.php?download=badips" 2>/dev/null | 
    grep -Ev "#|^ *$") http.log 

That's a little complex though, so let's break it down:

  • wget -O - https://zeustracker.abuse.ch/blocklist.php?download=badips 2>/dev/null | grep -Ev "#|^ *$" : this will download the list of IPs, remove any lines that contain # or are empty and print to standard output.
  • <(command) : This is called process substitution and is a trick that some shells (bash, zsh and ksh, but maybe others as well) have which enables you to treat the output of a process as an input file to another command.
  • So, the awk snipet sees two "files", the first is the output of the wget command piped trough grep and the second is the http.log file.
  • NR==FNR{a[$1]=NR; next} : NR is the current input line number and FNR is the same but of the current file. The two will be equal only while the first file is being read. This will, therefore, create the array a whose keys are the IPs and whose value is the current line number.
  • for(ip in a) : iterate over the a array, saving each of its keys (the IPs) as ip.
  • if($0~ip){} : if this line ($0) matches this ip.
  • print >> "line_"a[ip]"_ip.txt" : print the line into a file called line_, the number of the input line where this IP was seen and _ip.txt. The >> ensures that the file appended to and not overwritten.
0

Note that bro comes with bro-cut to extract data from its log files.

You could do:

bro-cut id.orig_h id.resp_h < conn.log | grep -wFf <(
  curl 'https://zeustracker.abuse.ch/blocklist.php?download=badips' |
  grep '^[^#]'
)

With awk:

awk '
  NR == FNR {
    if (/^[^#]/) bad[$0]=""
    next
  }
  ($1 in bad) || ($2 in bad)
' <(curl 'https://zeustracker.abuse.ch/blocklist.php?download=badips') \
  <(bro-cut id.orig_h id.resp_h < conn.log)

(more efficient as using hash tables instead of painstaking string matching in a loop).

Also note that the emerging threat rule-set has rules that track those Zeus tracker lists (and others). So if you use snort/surricata with that rule set in addition to bro-ids, doing it with bro would end up being redundant.

  • Sorry for the delayed response. I like the idea of grep -wFf <( curl 'zeustracker.abuse.ch/blocklist.php?download=badips' | grep '^[^#]' ) but I'm getting the error -sh: syntax error near unexpected token `(' Thoughts? – user112802 May 22 '15 at 3:29
  • Dumb question, how do I run multi-line awk commands? I've only ever used awk for single line stuff. – user112802 May 22 '15 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.