diffrence between ls -l *.<sometthing> | wc -l and ls -l | grep <something> | wc -l

Question

I have a folder with a huge file list that I want to count. I was doing ls -l *.json | wc -l and it was working fine, till the list apparently gone huge enough that this command didn't work any more.

I tried that later one ls -l | grep .json | wc -l and it worked like a charm.

So I want to know what is the difference between both? I think it's because the later does not ls all the file at once but rather streams the files continuously in the grep and then to wc? Is it so, or it works differently?

Answer 1

If you want to count the number of non-hidden files that have a .json extensions in the current directory, you'd do:

(){echo $#} *.json(NoN)

(N for nullglob, oN to disable sorting which we don't need here).

ls -l | grep .json | wc -l is wrong for a number of reasons:

• . is a regexp operator that match any single character. If you wanted to search for the .json string that would be grep -F .json or grep '\.json' or grep '[.]json'.
• That Xjson is looked for on each line. With ls -l, you're printing the file name, the user name, the group name, the target of symlinks all of which could contain Xjson.
• also file names may contain any byte value except 0 and that of / (link targets can contain /); that includes newline. So if you have a file called xjson\nyjson that is a symlink to ajson\nbjson, without -q, ls -l for that file will print 3 lines all of which will contain Xjson. You may also get surprises if some file names contain sequences of bytes that don't form valid characters in the current locale.
• grep | wc -l can generally be replaced with grep -c.

ls -l *.json | wc -l is even worse. Beside the arg list too long potential problem already noted by @L.ScottJohnson, there's also:

• if there's no non-hidden .json file, you'll get 0 but also an error from zsh as the *.json glob fails to match.
• For each of the arguments that are of type directory, ls -l lists their content, so if you have a dir.json directory, all the lines resulting from its listing will be counted. Generally, you want to use a -d when passing a glob expansion to ls.
• If any of the .json filename starts with -, that will be taken as an option by ls (especially the GNU or busybox implementations of ls which accept options even after non-option arguments).
• like above, you'll have problems if the filename or symlink target contains newlines.

You could fix most of those with something like:

LC_ALL=C ls -qd -- *.json | wc -l

But in there, all the actual job is done by the shell. The shell is the one which expands *.json to the list of matching files and passes it to ls. ls is only used there to print each on a separate line to feed to wc so it can count them. ls also does a lot of unnecessary work like doing a stat() system call on each to check it's there (with -l, it does a lstat() and a few uid/gid to user/group name resolutions, and readlink()s for symlinks), and sorts the list again (the shell already sorts the *.json expansion).

The shell is well able to count that expansion by itself.

With (){echo $#} *.json(NoN), we use an anonymous function, you could also use a temporary array: files=(*.json(NoN)); echo $#files.

Also note that it only needs to read the contents of the current directory to build that list, it doesn't need to try and look-up each file individually like ls does.

Note that that syntax is specific to zsh. The equivalent in a POSIX sh would be something like:

set -- [*].json *.json
case $1$2 in
  ('[*].json*.json') shift 2;;
  (*) shift;;
esac
echo "$#"

(and the list of files is in "$@" (sorted)).

Answer 2

Your original command might fail if the expanded list of files is too long, since it relies on the shell to expand the list of files.

ls -l | grep .json | wc -l 

Is OK because the command line is short (no expansion) and the grep does the filtering line by line instead.

Note that the dot matches any character, not just a dot. Use -F to indicate fixed strings:

ls -l | grep -F .json | wc -l 

so you don't accidentally match the file xjson or something similarly silly.

There are still caveats for this command as well, and other (more robust) ways to achieve your desired goal, but those issues aside, this is the difference between the two commands: filtering by filename globbing (i.e., shell expansion) vs. pipe-to-grep