Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I get the following error when trying to ls *.txt | wc -l a directory that contains many files:

-bash: /bin/ls: Argument list too long

Does the threshold of this "Argument list" dependent on distro or computer's spec? Usually, I'd pipe the result of such big result to some other commands (wc -l for example), so I'm not concerned with limits of the terminal.

share|improve this question
4  
That counts as parsing ls's output, which is a bad idea, so better avoid it. For counting see What's the best way to count the number of files in a directory?, for a tricky workaround see why for loop doesn't raise “argument too long” error?. –  manatwork May 18 '12 at 16:00
    
@manatwork Yes, I saw those questions, too. Just wondering a better way to use or redirect a long output from a command in a more general fashion. –  lacampane11a May 19 '12 at 3:01
add comment

4 Answers

up vote 7 down vote accepted

Your error message argument list too long comes from the * of ls *.txt.

This limits is a safety for both binary programs and your Kernel. You'll see on this page more information about it, and how it's used and computed.

There is no such limit on pipe size. So you can simply issue this command:

find -name '*.txt'  | wc -l

NB: On modern Linux, weird characters in filenames (like newlines) will be escaped with tools like ls or find, but still displayed from *. If you are on an old Unix, you'll need this command

find -name '*.txt' -exec echo \;  | wc -l

NB2: I was wondering how one can create a file with a newline in its name. It's not that hard, once you know the trick:

touch "hello
world"
share|improve this answer
1  
I modified it slightly to work in the case when there are filenames with newlines in them. You might also want to add a -maxdepth 1 if you are not intending to count files in subdirectories. –  Shawn J. Goff May 18 '12 at 17:25
    
You don't need the -exec echo \;. –  Mikel May 18 '12 at 18:57
    
@ShawnJ.Goff I have tested it. There's no need of ` echo` in current version of GNU find –  Coren May 18 '12 at 19:51
    
@Coren @Mikel - not everybody has GNU's find. The find on OS X and on busybox-based systems, and I would guess any BSD-based system shows the filename with a newline in it, which would mess with the count. –  Shawn J. Goff May 18 '12 at 21:01
    
Huh? wc -l is counting newlines. So we want it to have newlines. –  Mikel May 18 '12 at 22:05
show 3 more comments

It depends mainly on your version of the Linux kernel.

You should be able to see the limit for your system by running

getconf ARG_MAX

which tells you the maximum number of bytes a command line can have after being expanded by the shell.

In Linux < 2.6.23, the limit is usually 128 KB.

In Linux >= 2.6.25, the limit is either 128 KB, or 1/4 of your stack size (see ulimit -s), whichever is larger.

See the execve(2) man page for all the details.


Unfortunately, piping ls *.txt isn't going to fix the problem, because the limit is in the operating system, not the shell.

The shell expands the *.txt, then tries to call

exec("ls", "a.txt", "b.txt", ...)

and you have so many files matching *.txt that you're exceeding the 128 KB limit.

You'll have to do something like

find . -maxdepth 1 -name "*.txt" | wc -l

instead.

(And see Shawn J. Goff's comments below about file names that contain newlines.)

share|improve this answer
    
Sorry for not able to upvote an answer. Need more reputation. :( Thank you all!! –  lacampane11a May 19 '12 at 3:03
add comment

Another workaround:

ls | grep -c '\.txt$'

Even though ls produces more output than ls *.txt produces (or attempts to produce), it doesn't run into the "argument too long" problem, because you're not passing any arguments to ls. Note that grep takes a regular expression rather than a file matching pattern.

You might want to use:

ls -U | grep -c '\.txt$'

(assuming your version of ls supports this option). This tells ls not to sort its output, which could save both time and memory -- and in this case the order doesn't matter, since you're just counting files. The resources spent sorting the output are usually not significant, but in this case we already know you have a very large number of *.txt files.

And you should consider reorganizing your files so you don't have so many in a single directory. This may or may not be feasible.

share|improve this answer
add comment

MAX_ARG_PAGES appears to be a kernel parameter. Using find and xargs is a typical combination to address this limit but I'm not sure it'll work for wc.

Piping the output of find . -name \*\.txt to a file and counting the lines in that file should serve as a workaround.

share|improve this answer
    
You can do anything with ls's output, will not solve this. As long as the *.txt wildcard is expanded over the limit, will fail before even starting ls and generating any output. –  manatwork May 18 '12 at 16:05
    
True, I've updated my answer. –  Bram May 18 '12 at 16:11
    
Better. But to make it a replacement for ls you should specify -maxdepth 1 to avoid recursively scanning the subdirectories. –  manatwork May 18 '12 at 16:19
    
Sorry for not able to upvote an answer. Need more reputation. :( –  lacampane11a May 19 '12 at 3:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.