Search among the content of a large number of plain text files?

Question

I have a directory with 1,242,276 plain text files on my external hard drive. They are totally unorganized with meaningless numbers as their file names. The reason for that is because they were recovered by some recovery software after accidentally emptying a partition a while ago.

I am now under that directory and want to search for a word "polyhedron" among the content of all the files. I tried grep, but it fails:

$ grep polyhedron ./* 
bash: /bin/grep: Argument list too long

Is it because there are too many files under that directory? Also I would like to search for many other different keywords. I wonder what I can do now?

Answer 1

find . -type f -print0 | xargs -0 grep polyhedron

Given that you probably want to do something like copy the files with the matching the term into a similarly named folder...

find . -type f -print0 | xargs -0 grep -l polyhedron | while read i; do cp "$i" ../polyhedron; done

If you know there's no overlap between matches for terms (that is, no single file would have 'polyhedron' and some other term you want to organize by) then you could move mv instead of copy cp.

Answer 2

It's not that there are too many files, but that the list of arguments to the grep command is too long. It's a limitation of the execve(2) system call on the combined size of the list of arguments and list of environment variables passed along that call.

On Linux, since 2.6.23, it's an administrative limit that can be raised or lifted using ulimit -s (also sets the limit on the process stack size). So

ulimit -s unlimited

may work for you.

Otherwise, workarounds, most of which already mentioned in other answers consist in splitting the list of arguments so it fits under that limitation, or avoiding passing the list of files to execve.

ls | xargs grep polyhedron

(OK only because the file names only contain digits)

(it's xargs responsibility to split the list and run as many grep commands as necessary so the execve limitation is not reached).

find . -exec grep polyhedron {} +

Same, but this time, find does the splitting.

grep -r polyhedron .

(if your grep supports -r), this time, only 3 args of a few characters passed to grep, it's grep that builds the list of files internally and never passes it to a execve system call.

Some shells have builtin support for it.

With shells where grep is builtin, you wouldn't have the issue since builtins are not executed with a execve system call.

With ksh93, you can use:

command -x grep polyhedron *

And ksh93 will do the splitting.

zsh has the zargs command:

zargs * -- grep polyhedron

To search for more than one word, you can do:

grep -e word1 -e word2 ...

Or

grep 'word1
word2
...' ...

Or put the list of words in a file, one per line and use

grep -f that-file ...

Answer 3

Is it because there are too many files under that directory?

Yes. You use a wildcard expansion. That is all your filenames are expanded into the command line. This fails because there is a length limit. To determine this limit, try:

getconf ARG_MAX

Also I would like to search for many other different keywords. I wonder what I can do now?

Have you tried the grep recursive mode?

grep -r polyhedron .

---

There are also some other approaches as the other answers show. This article also gives some background on the problem and some more examples how to circumvent this length limitation.

I copied some of the examples here to give an idea:

using find:

find /nas/data/accounting/ -type f -exec ls -l {} \;

using xargs:

echo /nas/data/accounting/* | xargs ls -l

using a while loop:

find /nas/data/accounting/ -type f |
  while read file
  do
    mv /nas/data/accounting/$file /local/disk/
  done

Answer 4

That's precisely the case. There are workarounds for that problem.

Quick answer for the grep problem

find . -type f -exec grep -H polyhedron {} +

In the long term, I suggest you use a function to sort the files by size (for example) so you can work on smaller parts of your files. You can use this script:

#!/bin/sh
cd $1
files=`find . -maxdepth 1 -type f`
for i in $files; do
    folder=`du -k $i | cut -f1`
    mkdir $folder
    mv $i $folder/
done;

All you have to do is save it and run it with:

sh <name of the script> <folder with all your files>

Optionally you can add execution permissions to avoid the sh part.

Answer 5

You could try:

find . -print0|xargs -0 grep 'term1\|term2'

xargs will generate mulitple grep commands using its default maximum arguments. If you still get an "Argument list too long" error from grep you can use the --max-args option for xargs.
The -print0 and -0 make it use null terminated filenames to handle filenames with spaces.
You can search for multiple terms using \|.
There are a lot of files so you might want to look at ways to optimise grep.