Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I had a directory which had around 5 million files. When I tried to run the ls command from inside this directory, my system consumed a huge amount of memory and it hung after sometime. Is there an efficient way to list the files other than using the ls command?

share|improve this question
Make sure you don't have an alias for ls that uses --color or -F as that would mean doing a lstat(2) for each file. – Stéphane Chazelas Mar 17 '14 at 21:37
By the way, storing millions of files in a single directory is a rather bad idea. If you control the directory layout, perhaps split it by some criteria? – d33tah Mar 18 '14 at 1:21
Was that a pure ls call or did you use options? – Hauke Laging Mar 18 '14 at 3:27
@d33tah Yeah, 5 million is a lot! My root file system has a limit of 7 million inodes. – Mikel Mar 18 '14 at 14:05
5 million items to output -how are you looking at this - simple listing is too much to see - so what do you want the listing for? – Mark Mar 18 '14 at 15:43

Avoid sorting by

ls -f
ls -U
ls --sort=none
share|improve this answer
I wonder how much overhead the column layout adds, too. Adding the -1 flag could help. – Mikel Mar 18 '14 at 14:03
Probably not much, but every little bit helps, right? :) – Mikel Mar 18 '14 at 14:20
@Mikel Is that just a guess, or have you measured that? To me it seems that -1 takes even longer. – Hauke Laging Mar 18 '14 at 19:32
"-1" helps quite a bit. "ls -f -1" will avoid any stat calls and print everything immediately. The column output (which is the default when sending to a terminal) makes it buffer everything first. On my system, using btrfs in a directory with 8 million files (as created by "seq 1 8000000 | xargs touch"), "time ls -f -1 | wc -l" takes under 5 seconds, while "time ls -f -C | wc -l" takes over 30 seconds. – Scott Lamb Dec 15 '15 at 16:08
up vote 29 down vote accepted

ls actually sorts the files and tries to list them which becomes a huge overhead if we are trying to list more than a million files inside a directory. As mentioned in this link, we can use strace or find to list the files. However, those options also seemed unfeasible to my problem since I had 5 million files. After some bit of googling, I found that if we list the directories using getdents(), it is supposed to be faster, because ls, find and Python libraries use readdir() which is slower but uses getdents() underneath.

We can find the C code to list the files using getdents() from here:

 * List directories using getdents() because ls, find and Python libraries
 * use readdir() which is slower (but uses getdents() underneath.
 * Compile with 
 * ]$ gcc  getdents.c -o getdents
#define _GNU_SOURCE
#include <dirent.h>     /* Defines DT_* constants */
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/syscall.h>

#define handle_error(msg) \
       do { perror(msg); exit(EXIT_FAILURE); } while (0)

struct linux_dirent {
   long           d_ino;
   off_t          d_off;
   unsigned short d_reclen;
   char           d_name[];

#define BUF_SIZE 1024*1024*5

main(int argc, char *argv[])
   int fd, nread;
   char buf[BUF_SIZE];
   struct linux_dirent *d;
   int bpos;
   char d_type;

   fd = open(argc > 1 ? argv[1] : ".", O_RDONLY | O_DIRECTORY);
   if (fd == -1)

   for ( ; ; ) {
       nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
       if (nread == -1)

       if (nread == 0)

       for (bpos = 0; bpos < nread;) {
           d = (struct linux_dirent *) (buf + bpos);
           d_type = *(buf + bpos + d->d_reclen - 1);
           if( d->d_ino != 0 && d_type == DT_REG ) {
              printf("%s\n", (char *)d->d_name );
           bpos += d->d_reclen;


Copy the C program above into directory in which the files need to be listed. Then execute the below commands.

gcc  getdents.c -o getdents
share|improve this answer
Could you add a small benchmark in timing for which your case does display with ls? – Bernhard Mar 17 '14 at 16:34
Sweet. And you could add an option to simply count the entries (files) rather than listing their names (saving millions of calls to printf, for this listing). – ChuckCottrill Mar 17 '14 at 20:50
You know your directory is too big when you have to write custom code to list its contents... – casey Mar 18 '14 at 1:45
@casey Except you don't have to. All this talk about getdents vs readdir misses the point. – Mikel Mar 18 '14 at 13:35
Come on! It's already got 5 million files in there. Put your custom "ls" program into some other directory. – Johan Mar 19 '14 at 8:35

The most likely reason why it is slow is file type colouring, you can avoid this with \ls or /bin/ls turning off the colour options.

If you really have so many files in a dir, using find instead is also a good option.

share|improve this answer
I don't think this should have been downvoted. Sorting is one problem, but even without sorting, ls -U --color would take a long time since it would stat each file. So both are correct. – Mikel Mar 18 '14 at 13:59

I find that echo * works much faster than ls. YMMV.

share|improve this answer
The shell will sort the *. So this way is probably still very slow for 5 million files. – Mikel Mar 18 '14 at 13:52
@Mikel More than that, I'm pretty sure that 5 million files is over the point where globbing will break entirely. – evilsoup Mar 18 '14 at 15:26
Minimum file name length (for 5 million files) is 3 characters (maybe 4 if you stick to more common characters) plus delimiters = 4 chars per file, i.e 20 MB of command arguments. That is well over the common 2MB expanded command line length. Exec (and even the builtins) would baulk. – Johan Mar 19 '14 at 8:49

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.