0

I successfully used:

grep -wFf inputqueries.txt seachedfile.txt > results.txt

to search searchedfile.txt for each query in inputqueries.txt.

inputqueries.txt looks like:

213.183.56.186
216.176.100.240
216.215.112.149
217.23.49.178
222.29.197.232
23.235.201.32
23.253.150.120
202.112.166.5

searchedfile.txt looks like

168.68.129.127  184.73.191.34
199.133.78.171  202.112.166.5
64.180.139.190  199.141.121.11
199.133.186.162 128.118.250.5
54.145.167.92   168.68.129.73
199.154.229.66  23.75.15.164
162.79.16.103   199.134.135.69

and results.txt was correctly

199.133.78.171  202.112.166.5

Unfortunately, that is where my success stopped. When I put it to work in the real world, it didn't work. Every time it returned zero results.

I used the same inputqueries.txt as well as one with a query list of words (as opposed to IPs). Further, it's important to note that I do not have write privileges to the actual log file directories and most of the logs are zipped as .gz. Additionally, I'm trying to search multiple similar files at the same time (zcat http, zcat conn.*, etc)

zcat filestosearch.* | grep -wFf /home/username/inputqueries.txt > /home/username/results.txt

Didn't work (nor did it work if I took off -wF and left it just grep -f)

zgrep -wFf /home/username/inputqueries.txt filestosearch.* > /home/username/results.txt

Didn't work either.

The logs I'm searching in real life vary, but the http one looks like... (they are all bro logs) 

1432343999.435553       CuCcn04H20cc2ZHyEh      202.170.48.4    50501   197.138.26.55   80      4       GET     ndb.nal.usda.gov        /ndb/search/autosuggest?manu=&fgcd=&term=Coconut+milk       http://ndb.nal.usda.gov/ndb/foods?fgcd=&manu=&lfacet=&count=&max=35&sort=&qlookup=Oil%2C+palm&offset=&format=Abridged&new=&measureby=   Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36    0       12994   200     OK      -       -       -       (empty) -       -       -       -       -       FGGh0g4a24L8Q6CZUb     text/plain
1432343999.382108       CKPWGW2cubkRjFpTKf      197.166.19.125   63803   54.191.210.216  80      1       GET     client.ql2.com  /cc/diff/http.www.ars.usda.gov/_22Fpandp_22Flocations_22FcityPeopleList.cfm_23Fmodecode_23D60-64-05-10/20150409123538diff.html  -       WebTrends/3.0 (WinNT)  00       302     Moved Temporarily       -       -       -       (empty) -       -       -       -       -       -       -
1432343999.595036       Cz4XJl3uaq2Fxc0M9a      63.248.145.199  63004   197.155.76.112  80      1       GET     start2farm.gov  /sites/all/themes/contrib/twitter_bootstrap/images/arrow-green.png      http://start2farm.gov/  Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko     0       1498    200     OK      -       -       -       (empty) -       -       -       -       -       Fo69Ao3w36RxKcoH9f     image/png
1432343999.732470       CTPQZyQ7tX7BUjU5j       197.123.240.10    56863   216.58.217.132  80      36      GET     toolbarqueries.google.com      /tbr?client=navclient-auto&ch=63738508926&features=Rank&q=info:/url?q=http://www.ncbi.nlm.nih.gov/books/NBK8125/&sa=U&ei=FjjmVJriAceagwSM1oOIDg&ved=0CBsQFjAB&usg=AFQjCNHgMKW6EIWKxclKB9o-o21bQu7IOw     -       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618)    0       5928    403     Forbidden       -      --       (empty) -       -       -       -       -       F2UdRnxrFUEHJFdW4       text/html

Using the simplest single-line command that I can, how do I make the grep from a file work?

Improve this question
  • Are you sure that there are actual matches? Granted, you only showed a small sample, but I don't think any of those would be matched. Can you show us one that should be matched, but wasn't? – Eric Renouf May 23 '15 at 2:14
  • @EricRenouf I thought the same thing so I went back and grepped a known keyword based on a quick glance in the http.* and it worked fine. Further, I then added search queries that were simple words like the and portal to the inputqueries.txt, which should have given me a file at least a GB in size, but still 0 bytes for the results.txt. – user112802 May 23 '15 at 2:27
  • Did you create the input file on Windows? If there are hidden \r characters at the ends of the lines that could explain it. You might want to try to dos2unix it and see if that helps – Eric Renouf May 23 '15 at 2:33
  • The initial test query file with IPs was created in Windows (Notepad++). The subsequent query files with IPs and words have been created in both Notepad++ and VIM since I thought that might be an issue. Still doesn't help. I think it must be a syntax issue. Could the tab space in the actual files versus the single space in the successful initial search be the problem? – user112802 May 23 '15 at 2:36
  • So you actively converted using a tool like dos2unix? When I created a quick file and did your same experiment a \r caused the effects you're reporting. Vim wouldn't have automatically converted the file for you. – Eric Renouf May 23 '15 at 2:38
2

The problem seems to be that the file(s) were edited on Windows at some point, which added some extra \r characters to the ends, which aren't normally visible.

If you have the dos2unix command you can use that to convert the file. If you don't have that and there isn't any important whitespace at the ends of lines you can do it with GNU sed as follows:

sed -i -e 's/\s*$//' inputqueries.txt

to modify the file in place (the -i flag) and then replace any amount of whitespace at the end of the line with nothing, effectively deleting it. -i is not part of POSIX though, so if you need a portable solution you can use the rest of the sed command and redirect to a temporary file. When you're sure that file is right, rename it to the file you actually want.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.