3

I have a directory with 1,242,276 plain text files on my external hard drive. They are totally unorganized with meaningless numbers as their file names. The reason for that is because they were recovered by some recovery software after accidentally emptying a partition a while ago.

I am now under that directory and want to search for a word "polyhedron" among the content of all the files. I tried grep, but it fails:

$ grep polyhedron ./* 
bash: /bin/grep: Argument list too long

Is it because there are too many files under that directory? Also I would like to search for many other different keywords. I wonder what I can do now?

9

find . -type f -print0 | xargs -0 grep polyhedron

Given that you probably want to do something like copy the files with the matching the term into a similarly named folder...

find . -type f -print0 | xargs -0 grep -l polyhedron | while read i; do cp "$i" ../polyhedron; done

If you know there's no overlap between matches for terms (that is, no single file would have 'polyhedron' and some other term you want to organize by) then you could move mv instead of copy cp.

  • Thanks! And that extra moving found files into a directory is very thoughtful. Is it a copy or a move? I think it is a copy? – Tim Dec 15 '12 at 17:44
  • My example is a copy (cp) since it's unclear how to handle a single file that might match many terms. I'll update the answer for consistency between move/copy. – mghicks Dec 15 '12 at 17:48
  • 1
    Since you're already using GNU extensions you could as well go all the way with zeros: find . -type f -print0 | xargs -r0 grep -lZ polyhedron | xargs -r0 cp -t ../polyhedron – Stéphane Chazelas Dec 16 '12 at 7:50
7

It's not that there are too many files, but that the list of arguments to the grep command is too long. It's a limitation of the execve(2) system call on the combined size of the list of arguments and list of environment variables passed along that call.

On Linux, since 2.6.23, it's an administrative limit that can be raised or lifted using ulimit -s (also sets the limit on the process stack size). So

ulimit -s unlimited

may work for you.

Otherwise, workarounds, most of which already mentioned in other answers consist in splitting the list of arguments so it fits under that limitation, or avoiding passing the list of files to execve.

ls | xargs grep polyhedron

(OK only because the file names only contain digits)

(it's xargs responsibility to split the list and run as many grep commands as necessary so the execve limitation is not reached).

find . -exec grep polyhedron {} +

Same, but this time, find does the splitting.

grep -r polyhedron .

(if your grep supports -r), this time, only 3 args of a few characters passed to grep, it's grep that builds the list of files internally and never passes it to a execve system call.

Some shells have builtin support for it.

With shells where grep is builtin, you wouldn't have the issue since builtins are not executed with a execve system call.

With ksh93, you can use:

command -x grep polyhedron *

And ksh93 will do the splitting.

zsh has the zargs command:

zargs * -- grep polyhedron

To search for more than one word, you can do:

grep -e word1 -e word2 ...

Or

grep 'word1
word2
...' ...

Or put the list of words in a file, one per line and use

grep -f that-file ...
  • Thanks. ulimit -s is for RLIMIT_STACK, while ARG_MAX is limit on the number of arguments to execve() unix.stackexchange.com/a/58592/674. Are they different things correct? Which limit is the one that is relevant here? – Tim Jun 6 '18 at 17:55
5

Is it because there are too many files under that directory?

Yes. You use a wildcard expansion. That is all your filenames are expanded into the command line. This fails because there is a length limit. To determine this limit, try:

getconf ARG_MAX

Also I would like to search for many other different keywords. I wonder what I can do now?

Have you tried the grep recursive mode?

grep -r polyhedron .

There are also some other approaches as the other answers show. This article also gives some background on the problem and some more examples how to circumvent this length limitation.

I copied some of the examples here to give an idea:

using find:

find /nas/data/accounting/ -type f -exec ls -l {} \;

using xargs:

echo /nas/data/accounting/* | xargs ls -l

using a while loop:

find /nas/data/accounting/ -type f |
  while read file
  do
    mv /nas/data/accounting/$file /local/disk/
  done
  • to handle filenames with spaces: while IFS= read -r file – glenn jackman Dec 16 '12 at 2:11
4

That's precisely the case. There are workarounds for that problem.

Quick answer for the grep problem

find . -type f -exec grep -H polyhedron {} +

In the long term, I suggest you use a function to sort the files by size (for example) so you can work on smaller parts of your files. You can use this script:

#!/bin/sh
cd $1
files=`find . -maxdepth 1 -type f`
for i in $files; do
    folder=`du -k $i | cut -f1`
    mkdir $folder
    mv $i $folder/
done;

All you have to do is save it and run it with:

sh <name of the script> <folder with all your files>

Optionally you can add execution permissions to avoid the sh part.

2

You could try:

find . -print0|xargs -0 grep 'term1\|term2'

xargs will generate mulitple grep commands using its default maximum arguments. If you still get an "Argument list too long" error from grep you can use the --max-args option for xargs.
The -print0 and -0 make it use null terminated filenames to handle filenames with spaces.
You can search for multiple terms using \|.
There are a lot of files so you might want to look at ways to optimise grep.

  • 2
    Note that \| is not standard in grep (standard basic regular expressions don't have an alternation operator). grep -E 'term1|term2' is standard though, so is grep -e term1 -e term2 (no need for regular expressions here, so you can even use -F). – Stéphane Chazelas Dec 15 '12 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.