grep works with one filepath, not another

Question

I have two directories that both have a couple thousand files each, and I am trying to grep certain IPs from the files. My grep string is:

grep "IP" cdr/173/07/cdr_2018_07*

This grep string returns "grep: Argument list too long". However, when I do the following:

grep "IP" cdr/173/06/cdr_2018_06*

it returns what I am looking for.

Below is the ls -l for the parent directory for each of these. It seems that the difference is about 400KB, so I'm not sure that size is really the issue here. Am I missing something?

jeblin@debian:~$ ls -l cdr/173
total 18500
REDACTED
drwxr-xr-x 2 jeblin jeblin 2781184 Jul  2 09:34 06
drwxr-xr-x 2 jeblin jeblin 2826240 Aug  1 07:33 07

If it makes a difference, I wrote a Python script that automates this process (searching for multiple IPs), and it works for 06, but not 07 as well, which is why I tried to do the manual grep search first.

Answer 1

The shell is not able to call grep with too many files, or rather, the length of the command line¹ for calling an external utility has a limit, and you're hitting it when the shell tries to call grep with the the expanded cdr/173/07/cdr_2018_07* globbing pattern.

What you can do is either to grep each file individually, with

for pathname in cdr/173/07/cdr_2018_07*; do
    grep "IP" "$pathname" /dev/null
done

where the extra /dev/null will force grep to always report the filename of the file that matched, or you can use find:

find cdr/173/07 -maxdepth 1 -type f -name 'cdr_2018_07*' \
    -exec grep "IP" /dev/null {} +

which will be more efficient as grep will be called with as many matching pathnames as possible in batches.

It could also be that if you first cd into cdr/173/07 and do

grep "IP" cdr_2018_07*

it may work since the generated list of filenames would be shorter due to not containing the directory bits, but you're probably very close to the limit with 44.7k files and should seriously consider moving to another way of doing this, especially if you're expecting the number of files to fluctuate around that number.

Related:

• Understanding the -exec option of `find`
• What defines the maximum size for a command single argument? (tangentially related)

• How to cause "Argument list too long" error?

• Other questions on U&L relating to "Argument list too long"

---

¹_{The limit is on the combined length on the command line and the length of the environment (the sum of the length of each argument and environment variable's name and value, also accounting for the pointers to them), and it is a limit imposed by the execve() system call which is used by the shell to execute external commands. Built-in commands such as echo etc. do not have this issue.}

Answer 2

The issue is the maximum limit of bytes allowed in the shell command.

* is expanded to the total list of all files in the directory, so what matters isn't the file size but the length of the filenames and the amount of files.

You can get your machine's limit in bytes by running $ getconf ARG_MAX. Please note that this limit is imposed by the OS/Kernel and not by the shell itself.

A way to circumvent this is to use find:

$ find cdr/173/07/ -iname "cdr_2018_07*" -type f -exec grep "IP" {} \;

Answer 3

i am answering question on basis of below points:- There are two directories:- 1)cdr/173/07 2)cdr/173/06

There can be many files in these two directories from which i need to search

a)Below is first solution

grep -r "IP" cdr/173/07 -e "IP" cdr/173/06

b)But if there are many other directories you want to search you can use

grep -r "IP" cdr/173/*

c)Lets suppose we have 1000 files and we want to search only particular types of files. grep -r "IP" cdr/173/07/cdr_2018_07* -e grep "IP" cdr/173/06/cdr_2018_06*