I successfully used:
grep -wFf inputqueries.txt seachedfile.txt > results.txt
to search searchedfile.txt for each query in inputqueries.txt.
inputqueries.txt looks like:
220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168
searchedfile.txt looks like
22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199
and results.txt was correctly
Unfortunately, that is where my success stopped. When I put it to work in the real world, it didn't work. Every time it returned zero results.
I used the same inputqueries.txt as well as one with a query list of words (as opposed to IPs). Further, it's important to note that I do not have write privileges to the actual log file directories and most of the logs are zipped as .gz. Additionally, I'm trying to search multiple similar files at the same time (zcat http, zcat conn.*, etc)
zcat filestosearch.* | grep -wFf /home/username/inputqueries.txt > /home/username/results.txt
Didn't work (nor did it work if I took off -wF and left it just grep -f)
zgrep -wFf /home/username/inputqueries.txt filestosearch.* > /home/username/results.txt
Didn't work either.
The logs I'm searching in real life vary, but the http one looks like... (they are all bro logs)
1432343999.435553 CuCcn04H20cc2ZHyEh 188.8.131.52 50501 184.108.40.206 80 4 GET ndb.nal.usda.gov /ndb/search/autosuggest?manu=&fgcd=&term=Coconut+milk http://ndb.nal.usda.gov/ndb/foods?fgcd=&manu=&lfacet=&count=&max=35&sort=&qlookup=Oil%2C+palm&offset=&format=Abridged&new=&measureby= Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36 0 12994 200 OK - - - (empty) - - - - - FGGh0g4a24L8Q6CZUb text/plain 1432343999.382108 CKPWGW2cubkRjFpTKf 220.127.116.11 63803 18.104.22.168 80 1 GET client.ql2.com /cc/diff/http.www.ars.usda.gov/_22Fpandp_22Flocations_22FcityPeopleList.cfm_23Fmodecode_23D60-64-05-10/20150409123538diff.html - WebTrends/3.0 (WinNT) 00 302 Moved Temporarily - - - (empty) - - - - - - - 1432343999.595036 Cz4XJl3uaq2Fxc0M9a 22.214.171.124 63004 126.96.36.199 80 1 GET start2farm.gov /sites/all/themes/contrib/twitter_bootstrap/images/arrow-green.png http://start2farm.gov/ Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko 0 1498 200 OK - - - (empty) - - - - - Fo69Ao3w36RxKcoH9f image/png 1432343999.732470 CTPQZyQ7tX7BUjU5j 188.8.131.52 56863 184.108.40.206 80 36 GET toolbarqueries.google.com /tbr?client=navclient-auto&ch=63738508926&features=Rank&q=info:/url?q=http://www.ncbi.nlm.nih.gov/books/NBK8125/&sa=U&ei=FjjmVJriAceagwSM1oOIDg&ved=0CBsQFjAB&usg=AFQjCNHgMKW6EIWKxclKB9o-o21bQu7IOw - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618) 0 5928 403 Forbidden - -- (empty) - - - - - F2UdRnxrFUEHJFdW4 text/html
Using the simplest single-line command that I can, how do I make the grep from a file work?