Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a directory with 1,242,276 plain text files on my external hard drive. They are totally unorganized with meaningless numbers as their file names. The reason for that is because they were recovered by some recovery software after accidentally emptying a partition a while ago.

I am now under that directory and want to search for a word "polyhedron" among the content of all the files. I tried grep, but it fails:

$ grep polyhedron ./* 
bash: /bin/grep: Argument list too long

Is it because there are too many files under that directory? Also I would like to search for many other different keywords. I wonder what I can do now?

share|improve this question

5 Answers 5

You could try:

find . -print0|xargs -0 grep 'term1\|term2'

xargs will generate mulitple grep commands using its default maximum arguments. If you still get an "Argument list too long" error from grep you can use the --max-args option for xargs.
The -print0 and -0 make it use null terminated filenames to handle filenames with spaces.
You can search for multiple terms using \|.
There are a lot of files so you might want to look at ways to optimise grep.

share|improve this answer
2  
Note that \| is not standard in grep (standard basic regular expressions don't have an alternation operator). grep -E 'term1|term2' is standard though, so is grep -e term1 -e term2 (no need for regular expressions here, so you can even use -F). –  Stéphane Chazelas Dec 15 '12 at 22:38

find . -type f -print0 | xargs -0 grep polyhedron

Given that you probably want to do something like copy the files with the matching the term into a similarly named folder...

find . -type f -print0 | xargs -0 grep -l polyhedron | while read i; do cp "$i" ../polyhedron; done

If you know there's no overlap between matches for terms (that is, no single file would have 'polyhedron' and some other term you want to organize by) then you could move mv instead of copy cp.

share|improve this answer
    
Thanks! And that extra moving found files into a directory is very thoughtful. Is it a copy or a move? I think it is a copy? –  Tim Dec 15 '12 at 17:44
    
My example is a copy (cp) since it's unclear how to handle a single file that might match many terms. I'll update the answer for consistency between move/copy. –  mghicks Dec 15 '12 at 17:48
    
Since you're already using GNU extensions you could as well go all the way with zeros: find . -type f -print0 | xargs -r0 grep -lZ polyhedron | xargs -r0 cp -t ../polyhedron –  Stéphane Chazelas Dec 16 '12 at 7:50

That's precisely the case. There are workarounds for that problem.

Quick answer for the grep problem

find . -type f -exec grep -H polyhedron {} +

In the long term, I suggest you use a function to sort the files by size (for example) so you can work on smaller parts of your files. You can use this script:

#!/bin/sh
cd $1
files=`find . -maxdepth 1 -type f`
for i in $files; do
    folder=`du -k $i | cut -f1`
    mkdir $folder
    mv $i $folder/
done;

All you have to do is save it and run it with:

sh <name of the script> <folder with all your files>

Optionally you can add execution permissions to avoid the sh part.

share|improve this answer

Is it because there are too many files under that directory?

Yes. You use a wildcard expansion. That is all your filenames are expanded into the command line. This fails because there is a length limit. To determine this limit, try:

getconf ARG_MAX

Also I would like to search for many other different keywords. I wonder what I can do now?

Have you tried the grep recursive mode?

grep -r polyhedron .

There are also some other approaches as the other answers show. This article also gives some background on the problem and some more examples how to circumvent this length limitation.

I copied some of the examples here to give an idea:

using find:

find /nas/data/accounting/ -type f -exec ls -l {} \;

using xargs:

echo /nas/data/accounting/* | xargs ls -l

using a while loop:

find /nas/data/accounting/ -type f |
  while read file
  do
    mv /nas/data/accounting/$file /local/disk/
  done
share|improve this answer
    
to handle filenames with spaces: while IFS= read -r file –  glenn jackman Dec 16 '12 at 2:11

It's not that there are too many files, but that the list of arguments to the grep command is too long. It's a limitation of the execve(2) system call on the combined size of the list of arguments and list of environment variables passed along that call.

On Linux, since 2.6.23, it's an administrative limit that can be raised or lifted using ulimit -s (also sets the limit on the process stack size). So

ulimit -s unlimited

may work for you.

Otherwise, workarounds, most of which already mentioned in other answers consist in splitting the list of arguments so it fits under that limitation, or avoiding passing the list of files to execve.

ls | xargs grep polyhedron

(OK only because the file names only contain digits)

(it's xargs responsibility to split the list and run as many grep commands as necessary so the execve limitation is not reached).

find . -exec grep polyhedron {} +

Same, but this time, find does the splitting.

grep -r polyhedron .

(if your grep supports -r), this time, only 3 args of a few characters passed to grep, it's grep that builds the list of files internally and never passes it to a execve system call.

Some shells have builtin support for it.

With shells where grep is builtin, you wouldn't have the issue since builtins are not executed with a execve system call.

With ksh93, you can use:

command -x grep polyhedron *

And ksh93 will do the splitting.

zsh has the zargs command:

zargs * -- grep polyhedron

To search for more than one word, you can do:

grep -e word1 -e word2 ...

Or

grep 'word1
word2
...' ...

Or put the list of words in a file, one per line and use

grep -f that-file ...
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.