I successfully used:
grep -wFf inputqueries.txt seachedfile.txt > results.txt
to search searchedfile.txt for each query in inputqueries.txt.
inputqueries.txt looks like:
126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52
searchedfile.txt looks like
184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11
and results.txt was correctly
Unfortunately, that is where my success stopped. When I put it to work in the real world, it didn't work. Every time it returned zero results.
I used the same inputqueries.txt as well as one with a query list of words (as opposed to IPs). Further, it's important to note that I do not have write privileges to the actual log file directories and most of the logs are zipped as .gz. Additionally, I'm trying to search multiple similar files at the same time (zcat http, zcat conn.*, etc)
zcat filestosearch.* | grep -wFf /home/username/inputqueries.txt > /home/username/results.txt
Didn't work (nor did it work if I took off -wF and left it just grep -f)
zgrep -wFf /home/username/inputqueries.txt filestosearch.* > /home/username/results.txt
Didn't work either.
The logs I'm searching in real life vary, but the http one looks like... (they are all bro logs)
1432343999.435553 CuCcn04H20cc2ZHyEh 18.104.22.168 50501 22.214.171.124 80 4 GET ndb.nal.usda.gov /ndb/search/autosuggest?manu=&fgcd=&term=Coconut+milk http://ndb.nal.usda.gov/ndb/foods?fgcd=&manu=&lfacet=&count=&max=35&sort=&qlookup=Oil%2C+palm&offset=&format=Abridged&new=&measureby= Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.65 Safari/537.36 0 12994 200 OK - - - (empty) - - - - - FGGh0g4a24L8Q6CZUb text/plain 1432343999.382108 CKPWGW2cubkRjFpTKf 126.96.36.199 63803 188.8.131.52 80 1 GET client.ql2.com /cc/diff/http.www.ars.usda.gov/_22Fpandp_22Flocations_22FcityPeopleList.cfm_23Fmodecode_23D60-64-05-10/20150409123538diff.html - WebTrends/3.0 (WinNT) 00 302 Moved Temporarily - - - (empty) - - - - - - - 1432343999.595036 Cz4XJl3uaq2Fxc0M9a 184.108.40.206 63004 220.127.116.11 80 1 GET start2farm.gov /sites/all/themes/contrib/twitter_bootstrap/images/arrow-green.png http://start2farm.gov/ Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko 0 1498 200 OK - - - (empty) - - - - - Fo69Ao3w36RxKcoH9f image/png 1432343999.732470 CTPQZyQ7tX7BUjU5j 18.104.22.168 56863 22.214.171.124 80 36 GET toolbarqueries.google.com /tbr?client=navclient-auto&ch=63738508926&features=Rank&q=info:/url?q=http://www.ncbi.nlm.nih.gov/books/NBK8125/&sa=U&ei=FjjmVJriAceagwSM1oOIDg&ved=0CBsQFjAB&usg=AFQjCNHgMKW6EIWKxclKB9o-o21bQu7IOw - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.21022; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618) 0 5928 403 Forbidden - -- (empty) - - - - - F2UdRnxrFUEHJFdW4 text/html
Using the simplest single-line command that I can, how do I make the grep from a file work?