Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I have 119766 files in a folder. They are CSV files. I want to find out total number of lines of all files.
I'm trying to run following command:
cat * |wc -l
But the following error occurrs:
-bash: /bin/cat: Argument list too long
How can I do that? Is there any way around this?
One thing I would like to add that total number of lines would be very large.
If you want a line-count for each individual file:
find . -type f -exec wc -l {} + | awk '! /^[ 0-9]+[[:space:]]+total$/'
I've excluded the total lines because there will be several of them with this many files being processed. The find ... -exec ... + will try to fit as many filenames onto a single command line as possible, but that will be a LOT less than 119766 files....probably only several thousand (at most) per invocation of wc, and each one will result in its own independent 'total' line.
If you want the total number of lines in all files combined, here's one way of doing it:
find . -type f -exec wc -l {} + |
awk '/^[ 0-9]+[[:space:]]+total$/ {print $1}' |
xargs | sed -e 's/ /+/g' | bc
This prints only the line counts on the total lines, pipes that into xargs to get the counts all on one line, then sed to transform the spaces into + signs, and then pipes the lot into bc to do the calculation.
Example output:
$ cd /usr/share/doc
$ find . -type f -exec wc -l {} + |
awk '/^[ 0-9]+[[:space:]]+total$/ {print $1}' |
xargs | sed -e 's/ /+/g' | bc
53358931
awk '$2 == "total"' for an exact match rather than a regexp match. It will only "fail" on filenames that exactly match 'total', and there's really no way around that since wc doesn't have options to either exclude totals or print only totals. I've thought several times over the years that such options would be useful, but they don't exist.
– cas
Mar 1 '16 at 8:20
printf '1 total\n1 total foo\n' | awk '$2 == "total"' give you two lines.
– cuonglm
Mar 1 '16 at 8:23
/ total$/ then. which will still "fail" on filenames that are exactly 'total' but as i said there's no avoiding that.
– cas
Mar 1 '16 at 8:37
wc to other tools to remove the last line. But it still fail with file named foo\n99999 bar. It will add 999999 to your result.
– cuonglm
Mar 1 '16 at 8:40
Well, to give that cat from the question a new home, this should do:
find . -type f -exec cat {} + | wc -l
It executes a cat with the maximum acceptable number of filenames (+) again and again and pipes everything to wc. If you do not want to traverse subdirectories, a -maxdepth 1 has to be added to the find command, after the directory.
As an alternative, the --files0-from option to GNU wc could be used:
find . -type f -print0 | wc -l --files0-from=- | tail -1
This option makes wc read not the contents but the filenames from stdin, separated by null characters. With -print0, find will print those filenames null-byte separated. As wc will still print out line counts for every file, it is advisable to skip everything except the summary line at the end, hence the tail.
Both solutions have the advantage that they will work in any locale, whereas @cas' solutions have to be adapted ('total' is 'insgesamt' in German, e.g.).
/usr/bin/find: Argument list too long (ubuntu 18.04 bash). To clarify, it was searching for files in sub-directories, so like */*/s*.out
– arun
Aug 4 '20 at 22:24
*/*/s*.out, your shell will (try to) expand all matching filenames before even invoking find, which makes the whole find command kind of futile. It is the same problem as in the question. Something like find . -type f -name "s*.out" -exec ... will find all files matching s*.out in all subdirectories and do something with them. If you need a certain directory depth (e.g. exactly two subdir levels), play with -mindepth and -maxdepth options.
– Dubu
Aug 10 '20 at 14:00
If you have too many files, you probably don't want the information to go to stdout, so you can do something like this:
IFS="\n"; for file in find ./ -type f; do wc -l "$file" >> ~/linecount.txt; done
ls, cat, mvand other commands have this limitations. As the error already tells you, you are providing too many arguments to thecatcommand in this case. Usegetconf -a |grep MAX_ARGto see the MAX_ARG value that applies to your kernel. – Valentin Bajrami Mar 1 '16 at 8:17