I successfully used:
grep -wFf inputqueries.txt seachedfile.txt > results.txt
to search searchedfile.txt for each query in inputqueries.txt.
inputqueries.txt looks like:
213.183.56.186
216.176.100.240
216.215.112.149
217.23.49.178
222.29.197.232
23.235.201.32
23.253.150.120
202.112.166.5
searchedfile.txt looks like
168.68.129.127 184.73.191.34
199.133.78.171 202.112.166.5
64.180.139.190 199.141.121.11
199.133.186.162 128.118.250.5
54.145.167.92 168.68.129.73
199.154.229.66 23.75.15.164
162.79.16.103 199.134.135.69
and results.txt was correctly
199.133.78.171 202.112.166.5
Unfortunately, that is where my success stopped. When I put it to work in the real world, it didn't work. Every time it returned zero results.
I used the same inputqueries.txt as well as one with a query list of words (as opposed to IPs). Further, it's important to note that I do not have write privileges to the actual log file directories and most of the logs are zipped as .gz. Additionally, I'm trying to search multiple similar files at the same time (zcat http, zcat conn.*, etc)
zcat filestosearch.* | grep -wFf /home/username/inputqueries.txt > /home/username/results.txt
Didn't work (nor did it work if I took off -wF and left it just grep -f)
zgrep -wFf /home/username/inputqueries.txt filestosearch.* > /home/username/results.txt
Didn't work either.
The logs I'm searching in real life vary, but the http one looks like... (they are all bro logs)
1432343999.435553 CuCcn04H20cc2ZHyEh 202.170.48.4 50501 197.138.26.55 80 4 GET ndb.nal.usda.gov /ndb/search/autosuggest?manu=&fgcd=&term=Coconut+milk http://ndb.nal.usda.gov/ndb/foods?fgcd=&manu=&lfacet=&count=&max=35&sort=&qlookup=Oil%2C+palm&offset=&format=Abridged&new=&measureby= Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36 0 12994 200 OK - - - (empty) - - - - - FGGh0g4a24L8Q6CZUb text/plain
1432343999.382108 CKPWGW2cubkRjFpTKf 197.166.19.125 63803 54.191.210.216 80 1 GET client.ql2.com /cc/diff/http.www.ars.usda.gov/_22Fpandp_22Flocations_22FcityPeopleList.cfm_23Fmodecode_23D60-64-05-10/20150409123538diff.html - WebTrends/3.0 (WinNT) 00 302 Moved Temporarily - - - (empty) - - - - - - -
1432343999.595036 Cz4XJl3uaq2Fxc0M9a 63.248.145.199 63004 197.155.76.112 80 1 GET start2farm.gov /sites/all/themes/contrib/twitter_bootstrap/images/arrow-green.png http://start2farm.gov/ Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko 0 1498 200 OK - - - (empty) - - - - - Fo69Ao3w36RxKcoH9f image/png
1432343999.732470 CTPQZyQ7tX7BUjU5j 197.123.240.10 56863 216.58.217.132 80 36 GET toolbarqueries.google.com /tbr?client=navclient-auto&ch=63738508926&features=Rank&q=info:/url?q=http://www.ncbi.nlm.nih.gov/books/NBK8125/&sa=U&ei=FjjmVJriAceagwSM1oOIDg&ved=0CBsQFjAB&usg=AFQjCNHgMKW6EIWKxclKB9o-o21bQu7IOw - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618) 0 5928 403 Forbidden - -- (empty) - - - - - F2UdRnxrFUEHJFdW4 text/html
Using the simplest single-line command that I can, how do I make the grep from a file work?
\r
characters at the ends of the lines that could explain it. You might want to try todos2unix
it and see if that helpsdos2unix
? When I created a quick file and did your same experiment a\r
caused the effects you're reporting. Vim wouldn't have automatically converted the file for you.