Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

CentOS 5.9

I came across an issue the other day where a directory had a lot of files. To count it, I ran ls -l /foo/foo2/ | wc -l

Turns out that there were over 1 million files in a single directory (long story -- the root cause is getting fixed).

My question is: is there a faster way to do the count? What would be the most efficient way to get the count?

share|improve this question
5  
ls -l|wc -l would be off by one due to the total blocks in the first line of ls -l output –  Thomas Nyman Sep 10 '13 at 21:18
2  
@ThomasNyman It would actually be off by several because of the dot and dotdot pseudo entries, but those can be avoided by using the -A flag. -l is also problematic because of the reading file meta data in order to generate the extended list format. Forcing NOT -l by using \ls is a much better option (-1 is assumed when piping output.) See Gilles's answer for the best solution here. –  Caleb Sep 11 '13 at 9:29
1  
@Caleb ls -l doesn't output any hidden files nor the . and .. entries. ls -a output includes hidden files, including . and .. while ls -A output includes hidden files excluding . and ... In Gilles's answer the bash dotglob shell option causes the expansion to include hidden files excluding . and ... –  Thomas Nyman Sep 11 '13 at 9:45

12 Answers 12

up vote 24 down vote accepted

Short answer:

\ls -afq | wc -l

(This includes . and .., so subtract 2.)


When you list the files in a directory, three common things might happen:

  1. Enumerating the file names in the directory. This is inescapable: there is no way to count the files in a directory without enumerating them.
  2. Sorting the file names. Shell wildcards and the ls command do that.
  3. Calling stat to retrieve metadata about each directory entry, such as whether it is a directory.

#3 is the most expensive by far, because it requires loading an inode for each file. In comparison all the file names needed for #1 are compactly stored in a few blocks. #2 wastes some CPU time but it is often not a deal breaker.

If there are no newlines in file names, a simple ls -A | wc -l tells you how many files there are in the directory. Beware that if you have an alias for ls, this may trigger a call to stat (e.g. ls --color or ls -F need to know the file type, which requires a call to stat), so from the command line, call command ls -A | wc -l or \ls -A | wc -l to avoid an alias.

If there are newlines in the file name, whether newlines are listed or not depends on the Unix variant. GNU coreutils and BusyBox default to displaying ? for a newline, so they're safe.

Call ls -f to list the entries without sorting them (#2). This automatically turns on -a (at least on modern systems). The -f option is in POSIX but with optional status; most implementations support it, but not BusyBox. The option -q replaces non-printable characters including newlines by ?; it's POSIX but isn't supported by BusyBox, so omit it if you need BusyBox support at the expense of overcounting files whose name contains a newline character.

If the directory has no subdirectories, then most versions of find will not call stat on its entries (leaf directory optimization: a directory that has a link count of 2 cannot have subdirectories, so find doesn't need to look up the metadata of the entries unless a condition such as -type requires it). So find . | wc -l is a portable, fast way to count files in a directory provided that the directory has no subdirectories and that no file name contains a newline.

If the directory has no subdirectories but file names may contain newlines, try one of these (the second one should be faster if it's supported, but may not be noticeably so).

find -print0 | tr -dc \\0 | wc -c
find -printf a | wc -c

On the other hand, don't use find if the directory has subdirectories: even find . -maxdepth 1 calls stat on every entry (at least with GNU find and BusyBox find). You avoid sorting (#2) but you pay the price of an inode lookup (#3) which kills performance.

In the shell without external tools, you can run count the files in the current directory with set -- *; echo $#. This misses dot files (files whose name begins with .) and reports 1 instead of 0 in an empty directory. This is the fastest way to count files in small directories because it doesn't require starting an external program, but (except in zsh) wastes time for larger directories due to the sorting step (#2).

  • In bash, this is a reliable way to count the files in the current directory:

    shopt -s dotglob nullglob
    a=(*)
    echo ${#a[@]}
    
  • In ksh93, this is a reliable way to count the files in the current directory:

    FIGNORE='(.|..)'
    a=(~(N)*)
    echo ${#a[@]}
    
  • In zsh, this is a reliable way to count the files in the current directory:

    a=(*(DNoN))
    echo $#a
    

    If you have the mark_dirs option set, make sure to turn it off: a=(*(DNoN^M)).

  • In any POSIX shell, this is a reliable way to count the files in the current directory:

    total=0
    set -- *
    if [ $# -ne 1 ] || [ -e "$1" ] || [ -L "$1" ]; then total=$((total+$#)); fi
    set -- .[!.]*
    if [ $# -ne 1 ] || [ -e "$1" ] || [ -L "$1" ]; then total=$((total+$#)); fi
    set -- ..?*
    if [ $# -ne 1 ] || [ -e "$1" ] || [ -L "$1" ]; then total=$((total+$#)); fi
    echo "$total"
    

All of these methods sort the file names, except for the zsh one.

share|improve this answer
    
My empirical testing on >1 million files shows that find -maxdepth 1 easily keeps pace with \ls -U as long as you don't add anything like a -type declaration that has to do further checks. Are you sure GNU find actually calls stat? Even the slowdown on find -type is nothing compared to how much ls -l bogs if you make it return file details. On the other hand the clear speed winner is zsh using the non sorting glob. (sorted globs are 2x slower than ls while the non-sorting one is 2x faster). I wonder if file system types would significantly effect these results. –  Caleb Sep 11 '13 at 9:44
    
@Caleb I ran strace. This is only true if the directory has subdirectories: otherwise find's leaf directory optimization kicks in (even without -maxdepth 1), I should have mentioned that. A lot of things can affect the result, including the filesystem type (calling stat is a lot more expensive on filesystems that represent directories as linear lists than on filesystems that represent directories as trees), whether the inodes were all created together and are thus close by on the disk, cold or hot cache, etc. –  Gilles Sep 11 '13 at 9:55
1  
Historically, ls -f has been the reliable way to prevent calling stat - this is often simply described today as "output is not sorted" (which it also causes), and does include . and ... -A and -U are not standard options. –  Random832 Sep 11 '13 at 12:59
find /foo/foo2/ -maxdepth 1 | wc -l

Is considerably faster on my machine but the local . directory is added to the count.

share|improve this answer
    
Thanks. I'm compelled to ask a silly question though: why is it faster? Because it's not bothering to look-up file attributes? –  Mike B Sep 10 '13 at 20:42
2  
Yes, that's my understanding. As long as your not using the -type parameter find should be faster than ls –  Joel Taylor Sep 10 '13 at 21:02
    
Hmmm.... if I'm understanding the documentation of find well, this should actually be better than my answer. Anyone with more experience can verify? –  Luis Machuca Sep 11 '13 at 2:38

ls -1U before the pipe should spend just a bit less resources, as it does no attempt to sort the file entries, it just reads them as they are sorted in the folder on disk. It also produces less output, meaning slightly less work for wc.

You could also use ls -f which is more or less a shortcut for ls -1aU.

Dunno if there is a resource-efficient way to do it via a command without piping though.

share|improve this answer
8  
Btw, -1 is implied when the output goes to a pipe –  enzotib Sep 10 '13 at 21:04
    
@enzotib - it is? Wow... one learns something new every day! –  Luis Machuca Sep 10 '13 at 21:25

Another point of comparison. While not being a shell oneliner, this C program doesn't do anything superflous. Note that hidden files are ignored to match the output of ls|wc -l (ls -l|wc -l is off by one due to the total blocks in the first line of output).

#include <stdio.h>
#include <stdlib.h>
#include <dirent.h>
#include <error.h>
#include <errno.h>

int main(int argc, char *argv[])
{
    int file_count = 0;
    DIR * dirp;
    struct dirent * entry;

    if (argc < 2)
        error(EXIT_FAILURE, 0, "missing argument");

    if(!(dirp = opendir(argv[1])))
        error(EXIT_FAILURE, errno, "could not open '%s'", argv[1]);

    while ((entry = readdir(dirp)) != NULL) {
        if (entry->d_name[0] == '.') { /* ignore hidden files */
            continue;
        }
        file_count++;
    }
    closedir(dirp);

    printf("%d\n", file_count);
}
share|improve this answer
    
Absolutely nothing. I'm using the SE Android alpha client and it's got a crazy bug that is screwing up my comments; some are being dropped and some are being posted on the wrong answers! Sorry for the confusion. –  Caleb Sep 11 '13 at 9:04

You could try perl -e 'opendir($dh,".");$i=0;while(readdir $dh){$i++};print "$i\n";'

It'd be interesting to compare timings with your shell pipe.

share|improve this answer
    
On my tests, this keeps pretty much exactly the same pace as the three other fastest solutions (find -maxdepth 1 | wc -l, \ls -AU | wc -l and the zsh based non sorting glob and array count). In other words it beats out the options with various inefficiencies such as sorting or reading extraneous file properties. I would venture to say since it doesn't earn you anything either, it isn't worth using over a simpler solution unless you happen to be in perl already :) –  Caleb Sep 11 '13 at 9:53
    
Note that this will include the . and .. directory entries in the count, so you need to subtract two to get the actual number of files (and subdirectories). In modern Perl, perl -E 'opendir $dh, "."; $i++ while readdir $dh; say $i - 2' would do it. –  Ilmari Karonen Sep 11 '13 at 10:36

A bash-only solution, not requiring any external program, but don't know how much efficient:

list=(*)
echo "${#list[@]}"
share|improve this answer
    
Glob expansion isn't necessary the most resource efficient way to do this. Besides most shells having an upper limit to the number of items they will even process so this will probably bomb when dealing with a million plus items, it also sorts the output. The solutions involving find or ls without sorting options will be faster. –  Caleb Sep 11 '13 at 6:37

After fixing the issue from @Joel 's answer, where it added . as a file:

find /foo/foo2 -maxdepth 1 | tail -n +2 | wc -l

tail simply removes the first line, meaning that . isn't counted anymore.

share|improve this answer
    
Adding a pair of pipes in order to omit one line of wc input is not very efficient as the overhead increases linearly with regard to input size. In this case, why not simply decrement the final count to compensate for it being off by one, which is a constant time operation: echo $(( $(find /foo/foo2 -maxdepth 1 | wc -l) - 1)) –  Thomas Nyman Sep 11 '13 at 6:32
    
Rather than feed that much data through another process, it would probably be better to just do some math on the final output. let count = $(find /foo/foo2 -maxdepth 1 | wc -l) - 2 –  Caleb Sep 11 '13 at 6:34

os.listdir() in python can do the work for you. It gives an array of the contents of the directory, excluding the special '.' and '..' files. Also, no need to worry abt files with special characters like '\n' in the name.

python -c 'import os;print len(os.listdir("."))'

following is the time taken by the above python command compared with the 'ls -Af' command.

~/test$ time ls -Af |wc -l
399144

real    0m0.300s
user    0m0.104s
sys     0m0.240s
~/test$ time python -c 'import os;print len(os.listdir("."))'
399142

real    0m0.249s
user    0m0.064s
sys     0m0.180s
share|improve this answer

ls -1 | wc -l comes immediately to my mind. Whether ls -1U is faster than ls -1 is purely academic - the difference should be negligible but for very large directories.

share|improve this answer

From this answer, I can think of this one as a possible solution.

/*
 * List directories using getdents() because ls, find and Python libraries
 * use readdir() which is slower (but uses getdents() underneath.
 *
 * Compile with 
 * ]$ gcc  getdents.c -o getdents
 */
#define _GNU_SOURCE
#include <dirent.h>     /* Defines DT_* constants */
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/syscall.h>

#define handle_error(msg) \
       do { perror(msg); exit(EXIT_FAILURE); } while (0)

struct linux_dirent {
   long           d_ino;
   off_t          d_off;
   unsigned short d_reclen;
   char           d_name[];
};

#define BUF_SIZE 1024*1024*5

int
main(int argc, char *argv[])
{
   int fd, nread;
   char buf[BUF_SIZE];
   struct linux_dirent *d;
   int bpos;
   char d_type;

   fd = open(argc > 1 ? argv[1] : ".", O_RDONLY | O_DIRECTORY);
   if (fd == -1)
       handle_error("open");

   for ( ; ; ) {
       nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
       if (nread == -1)
           handle_error("getdents");

       if (nread == 0)
           break;

       for (bpos = 0; bpos < nread;) {
           d = (struct linux_dirent *) (buf + bpos);
           d_type = *(buf + bpos + d->d_reclen - 1);
           if( d->d_ino != 0 && d_type == DT_REG ) {
              printf("%s\n", (char *)d->d_name );
           }
           bpos += d->d_reclen;
       }
   }

   exit(EXIT_SUCCESS);
}

Copy the C program above into directory in which the files need to be listed. Then execute the below commands.

gcc  getdents.c -o getdents
./getdents | wc -l
share|improve this answer

Probably the most resource efficient way would involve no outside process invocations. So I'd wager on...

cglb() ( c=0 ; set --
    tglb() { [ -e "$2" ] || [ -L "$2" ] &&
       c=$(($c+$#-1))
    }
    for glb in '.?*' \*
    do  tglb $1 ${glb##.*} ${glb#\*}
        set -- ..
    done
    echo $c
)
share|improve this answer

I would think echo * would be more efficient than any 'ls' command:

echo * | wc -w
share|improve this answer
4  
What about files with a space in their name? echo 'Hello World'|wc -w produces 2. –  Joseph R. Sep 11 '13 at 20:52
    
@JosephR. Caveat Emptor –  Dan Garthwaite Sep 12 '13 at 0:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.