I successfully used:
grep -wFf inputqueries.txt seachedfile.txt > results.txt
to search searchedfile.txt for each query in inputqueries.txt.
inputqueries.txt looks like:
18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124
searchedfile.txt looks like
126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52
and results.txt was correctly
Unfortunately, that is where my success stopped. When I put it to work in the real world, it didn't work. Every time it returned zero results.
I used the same inputqueries.txt as well as one with a query list of words (as opposed to IPs). Further, it's important to note that I do not have write privileges to the actual log file directories and most of the logs are zipped as .gz. Additionally, I'm trying to search multiple similar files at the same time (zcat http, zcat conn.*, etc)
zcat filestosearch.* | grep -wFf /home/username/inputqueries.txt > /home/username/results.txt
Didn't work (nor did it work if I took off -wF and left it just grep -f)
zgrep -wFf /home/username/inputqueries.txt filestosearch.* > /home/username/results.txt
Didn't work either.
The logs I'm searching in real life vary, but the http one looks like... (they are all bro logs)
1432343999.435553 CuCcn04H20cc2ZHyEh 184.108.40.206 50501 220.127.116.11 80 4 GET ndb.nal.usda.gov /ndb/search/autosuggest?manu=&fgcd=&term=Coconut+milk http://ndb.nal.usda.gov/ndb/foods?fgcd=&manu=&lfacet=&count=&max=35&sort=&qlookup=Oil%2C+palm&offset=&format=Abridged&new=&measureby= Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36 0 12994 200 OK - - - (empty) - - - - - FGGh0g4a24L8Q6CZUb text/plain 1432343999.382108 CKPWGW2cubkRjFpTKf 18.104.22.168 63803 22.214.171.124 80 1 GET client.ql2.com /cc/diff/http.www.ars.usda.gov/_22Fpandp_22Flocations_22FcityPeopleList.cfm_23Fmodecode_23D60-64-05-10/20150409123538diff.html - WebTrends/3.0 (WinNT) 00 302 Moved Temporarily - - - (empty) - - - - - - - 1432343999.595036 Cz4XJl3uaq2Fxc0M9a 126.96.36.199 63004 188.8.131.52 80 1 GET start2farm.gov /sites/all/themes/contrib/twitter_bootstrap/images/arrow-green.png http://start2farm.gov/ Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko 0 1498 200 OK - - - (empty) - - - - - Fo69Ao3w36RxKcoH9f image/png 1432343999.732470 CTPQZyQ7tX7BUjU5j 184.108.40.206 56863 220.127.116.11 80 36 GET toolbarqueries.google.com /tbr?client=navclient-auto&ch=63738508926&features=Rank&q=info:/url?q=http://www.ncbi.nlm.nih.gov/books/NBK8125/&sa=U&ei=FjjmVJriAceagwSM1oOIDg&ved=0CBsQFjAB&usg=AFQjCNHgMKW6EIWKxclKB9o-o21bQu7IOw - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618) 0 5928 403 Forbidden - -- (empty) - - - - - F2UdRnxrFUEHJFdW4 text/html
Using the simplest single-line command that I can, how do I make the grep from a file work?