1

I have a directory with 2000+ text files. I am trying to make a script that:

  1. Reads a list of IP addresses from ip.txt
  2. Cats each file in the directory
  3. Greps each file for the IP address

If keyword is found, echoes the keyword and the file name to a file.

The output should be like this:

$ cat
results.txt
192.168.2.3 was found in 23233.txt
192.168.4.0 was found in 2323.txt

At the moment I have this:

while read p; do
for filename in *.txt; do
if cat $filename | grep "$p" 
then echo "$p" is  "$filename" | tee result.txt
fi
done
done<ips.txt

However this also echoes all file names into the results. How can I fix this?

4

First, save a cat by not using one when you don't need it. Rather than:

cat haystack | grep needle

You can simply:

grep needle haystack

As for your script:

> results.txt  # start with a fresh file for every run
while read ip; do
    grep "$ip" *  | grep -Ev 'results\.txt|ips\.txt' >> results.txt
done < ips.txt

The grep-into-grep pipeline is to prevent adding entries from the input and output files into the output file.

If you have a zillion files to check and you're getting argument list too long, we can use a tool like xargs to break our command up into chunks short enough for the shell to permit:

> results.txt  # start with a fresh file for every run
while read ip; do
    find . -type f -maxdepth 1 -not -name ips.txt -not -name results.txt -print0 | xargs -0 grep "$ip" >> results.txt
done < ips.txt

Here we're filtering out the input and output files with logic fed into find, so we no longer need to grep into grep.

  • Thanks, i am getting the error 'grep argument list too long' this is why i originally did a loop. ( i should of mentioned) – jonny b Apr 15 at 16:09
  • Solution that can handle a zillion files has been provided. – DopeGhoti Apr 15 at 16:17
0

Assuming your file only has IPv4 addresses (no IPv6), you can just run something like the following:

find [dir1] -maxdepth 1 -type f -iname ip.txt -exec grep -H '[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*' {} \;

This should handle an unlimited quantity of files. Your output will look something like [dir1]/ip.txt:1.2.3.4. Of course, you'll need to replace "[dir1]" with an actual directory (or you can omit this option, in which case find will just use your current working directory). You can specify multiple directories if you wish. In fact, there shouldn't be any real limit as to how many directories you can put in the find command, as long as you put them before the "-maxdepth" argument.

0

Another option, if you're not married to grep for this, is my favorite tool I've adopted to replace grep long ago - ack. It can pretty much drop into your provided script and provide the output you need. (It is searching recursively by default, worth noting if you don't want files in subfolders searched).

while read p; do
  for filename in $(ack -l $p); do
    echo "$p found in $filename" >> results.txt
  done
done<ips.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.