3

I need to select specific data's from log files. I need two scripts:

  1. I need to select all IP addresses that only visited /page1
  2. I need to select all IP addresses that visited /page1 but never visited /page2

I have my desired logs in a .tar file. I want them extracted into a folder, and then I will use the script to parse them and delete them. ALL duplicated IP addresses.

This is what I have so far:

# filter /page1 visitors
cat access.log | grep "/page1" > /tmp/res.txt
# take the IP portion of record
cat res.txt | grep '^[[:alnum:]]*\.[[:alnum:]]*\.[[:alnum:]]*\.[[:alnum:]]*' -o > result.txt

Typical access log looks like

162.158.86.83 - - [22/May/2016:06:31:18 -0400] "GET /page1?vtid=nb3 HTTP/1.1" 301 128 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
6
  • Can you explain what you have tried, please? – Kevdog777 May 26 '16 at 13:10
  • I did this filter /page visitors cat access.log | grep /page1 > /tmp/res.txt And # take the IP portion of record cat res.txt | grep '^[[:alnum:]]*\.[[:alnum:]]*\.[[:alnum:]]*\.[[:alnum:]]*' -o > result.txt – Delirium May 26 '16 at 13:11
  • Please edit your question and show us an example of your input and your desired output. Also, include what you tried in the question itself, not in the comments. Comments are hard to read and easy to miss and can be deleted without warning. – terdon May 26 '16 at 13:20
  • Thanks for the edit but we need to see an example of your input and your desired output. We can't help you parse something unless you tell us what we're parsing. – terdon May 26 '16 at 13:37
  • I tried to edit but since your question is not clear, I'm not sure I did a good job. What does "ALL duplicated IP addresses."? Do you want to print all duplicates? Do you want to remove all duplicates? – terdon May 26 '16 at 13:43
2
awk '/^\/page1?/ {print $1}' /path/to/access.log | sort -u > result.txt

If you want a count of each unique IP, change sort -u to sort | uniq -c

If you want to match only the request-path field of the log (rather than the entire line) against /page1:

awk '$7 ~ /^\/page1?/ {print $1}' /path/to/access.log | sort -u > result.txt

Note: I think nginx access logs are the same as apache access logs. If not, count the fields (count every space, including the one between the the Date:Time and the TimeZone) in the nginx log, and use the correct field number instead of $7

Finally, if you want to print both the IP address (or hostname if they've already been resolved) and the request path:

awk -v OFS='\t' '$7 ~ /^\/page1?/ {print $1, $7}' /path/to/access.log |
    sort -u > result.txt

To see IP addresses that have visited /page1 but have never visited /page2:

awk '$7 ~ /^\/page1?/ {print $1}' /path/to/access.log | sort -u > result1.txt
awk '$7 ~ /^\/page2?/ {print $1}' /path/to/access.log | sort -u > result2.txt
comm -2 -3 result1.txt result2.txt

comm's -2 option suppresses lines that appear only in result2.txt, and -3 suppresses lines that appear in both files. output is thus lines that appear only in results1.txt.

see man comm for more details.

7
  • And how I will know this IPs was not on /page2 and how I will delete duplicates? Is that possible in same time when is proceded awk command? – Delirium May 26 '16 at 13:30
  • 1. the awk script ignores everything but matching lines. /page2 doesn't match /page1 so it will be ignored. 2. i didn't notice the no-dupes part of your question, i'll update my answer. – cas May 26 '16 at 13:37
  • the script WILL, however, match /page10, /page11, /page100 and so on (exactly as your grep command does). I'd have to know more about your URL paths before I could come up with a reliable regexp to exclude them too. i.e. what normally comes immediately after /page1 in your logs. is /page the beginning of the request path? please add some example log lines to your question. – cas May 26 '16 at 13:43
  • Access log for /page1 looks like 162.158.86.83 - - [22/May/2016:06:31:18 -0400] "GET /page1?vtid=nb3 HTTP/1.1" 301 128 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0" I don't know if "awk" command will work even there are some arguments in that URL. – Delirium May 26 '16 at 13:45
  • awk has no problem with URL args. btw, don't post long examples or code in comments. edit your question and add them there. – cas May 26 '16 at 13:53
0
  • Create a sorted list of IPs that visited Page1
  • Create a sorted list of IPs that visited Page2
  • Use "diff" on the two lists to find those that visited one page without visiting the other (the '>' or '<' symbol at the beginning on the list distinguishes those from Page1 from those of Page2)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.