70

I had a directory which had around 5 million files. When I tried to run the ls command from inside this directory, my system consumed a huge amount of memory and it hung after sometime. Is there an efficient way to list the files other than using the ls command?

  • 11
    Make sure you don't have an alias for ls that uses --color or -F as that would mean doing a lstat(2) for each file. – Stéphane Chazelas Mar 17 '14 at 21:37
  • 4
    By the way, storing millions of files in a single directory is a rather bad idea. If you control the directory layout, perhaps split it by some criteria? – d33tah Mar 18 '14 at 1:21
  • Was that a pure ls call or did you use options? – Hauke Laging Mar 18 '14 at 3:27
  • 1
    @d33tah Yeah, 5 million is a lot! My root file system has a limit of 7 million inodes. – Mikel Mar 18 '14 at 14:05
  • 7
    5 million items to output -how are you looking at this - simple listing is too much to see - so what do you want the listing for? – user151019 Mar 18 '14 at 15:43
66

Avoid sorting by using:

ls --sort=none # "do not sort; list entries in directory order"

Or, equivalently:

ls -U
  • 10
    I wonder how much overhead the column layout adds, too. Adding the -1 flag could help. – Mikel Mar 18 '14 at 14:03
  • Probably not much, but every little bit helps, right? :) – Mikel Mar 18 '14 at 14:20
  • 1
    @Mikel Is that just a guess, or have you measured that? To me it seems that -1 takes even longer. – Hauke Laging Mar 18 '14 at 19:32
  • 10
    "-1" helps quite a bit. "ls -f -1" will avoid any stat calls and print everything immediately. The column output (which is the default when sending to a terminal) makes it buffer everything first. On my system, using btrfs in a directory with 8 million files (as created by "seq 1 8000000 | xargs touch"), "time ls -f -1 | wc -l" takes under 5 seconds, while "time ls -f -C | wc -l" takes over 30 seconds. – Scott Lamb Dec 15 '15 at 16:08
  • 1
    @ToolmakerSteve The default behavior (-C when stdout is a terminal, -1 when it's a pipe) is confusing. When you're experimenting and measuring, you flip between seeing the output (to ensure the command is doing what you expect) and suppressing it (to avoid the confounding factor of the terminal application's throughput). Better to use commands that behave in the same way in both modes, so explicitly define the output format via -1, -C, -l, etc. – Scott Lamb Apr 1 at 16:17
47

ls actually sorts the files and tries to list them which becomes a huge overhead if we are trying to list more than a million files inside a directory. As mentioned in this link, we can use strace or find to list the files. However, those options also seemed unfeasible to my problem since I had 5 million files. After some bit of googling, I found that if we list the directories using getdents(), it is supposed to be faster, because ls, find and Python libraries use readdir() which is slower but uses getdents() underneath.

We can find the C code to list the files using getdents() from here:

/*
 * List directories using getdents() because ls, find and Python libraries
 * use readdir() which is slower (but uses getdents() underneath.
 *
 * Compile with 
 * ]$ gcc  getdents.c -o getdents
 */
#define _GNU_SOURCE
#include <dirent.h>     /* Defines DT_* constants */
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/syscall.h>

#define handle_error(msg) \
       do { perror(msg); exit(EXIT_FAILURE); } while (0)

struct linux_dirent {
   long           d_ino;
   off_t          d_off;
   unsigned short d_reclen;
   char           d_name[];
};

#define BUF_SIZE 1024*1024*5

int
main(int argc, char *argv[])
{
   int fd, nread;
   char buf[BUF_SIZE];
   struct linux_dirent *d;
   int bpos;
   char d_type;

   fd = open(argc > 1 ? argv[1] : ".", O_RDONLY | O_DIRECTORY);
   if (fd == -1)
       handle_error("open");

   for ( ; ; ) {
       nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
       if (nread == -1)
           handle_error("getdents");

       if (nread == 0)
           break;

       for (bpos = 0; bpos < nread;) {
           d = (struct linux_dirent *) (buf + bpos);
           d_type = *(buf + bpos + d->d_reclen - 1);
           if( d->d_ino != 0 && d_type == DT_REG ) {
              printf("%s\n", (char *)d->d_name );
           }
           bpos += d->d_reclen;
       }
   }

   exit(EXIT_SUCCESS);
}

Copy the C program above into directory in which the files need to be listed. Then execute the below commands.

gcc  getdents.c -o getdents
./getdents

Timings example: getdents can be much faster than ls -f, depending on the system configuration. Here are some timings demonstrating a 40x speed increase for listing a directory containing about 500k files over an NFS mount in a compute cluster. Each command was run 10 times in immediate succession, first getdents, then ls -f. The first run is significantly slower than all others, probably due to NFS caching page faults. (Aside: over this mount, the d_type field is unreliable, in the sense that many files appear as "unknown" type.)

command: getdents $bigdir
usr:0.08 sys:0.96  wall:280.79 CPU:0%
usr:0.06 sys:0.18  wall:0.25   CPU:97%
usr:0.05 sys:0.16  wall:0.21   CPU:99%
usr:0.04 sys:0.18  wall:0.23   CPU:98%
usr:0.05 sys:0.20  wall:0.26   CPU:99%
usr:0.04 sys:0.18  wall:0.22   CPU:99%
usr:0.04 sys:0.17  wall:0.22   CPU:99%
usr:0.04 sys:0.20  wall:0.25   CPU:99%
usr:0.06 sys:0.18  wall:0.25   CPU:98%
usr:0.06 sys:0.18  wall:0.25   CPU:98%
command: /bin/ls -f $bigdir
usr:0.53 sys:8.39  wall:8.97   CPU:99%
usr:0.53 sys:7.65  wall:8.20   CPU:99%
usr:0.44 sys:7.91  wall:8.36   CPU:99%
usr:0.50 sys:8.00  wall:8.51   CPU:100%
usr:0.41 sys:7.73  wall:8.15   CPU:99%
usr:0.47 sys:8.84  wall:9.32   CPU:99%
usr:0.57 sys:9.78  wall:10.36  CPU:99%
usr:0.53 sys:10.75 wall:11.29  CPU:99%
usr:0.46 sys:8.76  wall:9.25   CPU:99%
usr:0.50 sys:8.58  wall:9.13   CPU:99%
  • 14
    Could you add a small benchmark in timing for which your case does display with ls? – Bernhard Mar 17 '14 at 16:34
  • 1
    Sweet. And you could add an option to simply count the entries (files) rather than listing their names (saving millions of calls to printf, for this listing). – ChuckCottrill Mar 17 '14 at 20:50
  • 29
    You know your directory is too big when you have to write custom code to list its contents... – casey Mar 18 '14 at 1:45
  • 1
    @casey Except you don't have to. All this talk about getdents vs readdir misses the point. – Mikel Mar 18 '14 at 13:35
  • 9
    Come on! It's already got 5 million files in there. Put your custom "ls" program into some other directory. – Johan Mar 19 '14 at 8:35
12

The most likely reason why it is slow is file type colouring, you can avoid this with \ls or /bin/ls turning off the colour options.

If you really have so many files in a dir, using find instead is also a good option.

  • 7
    I don't think this should have been downvoted. Sorting is one problem, but even without sorting, ls -U --color would take a long time since it would stat each file. So both are correct. – Mikel Mar 18 '14 at 13:59
  • Turning coloring off has a huge impact on the performance of ls and it is aliased by default in many many .bashrcs out there. – Victor Schröder Aug 7 '18 at 13:59
  • Yup I did a /bin/ls -U and got output in no time, compared to waiting for a very long time before – khebbie Oct 11 at 7:03
-3

I find that echo * works much faster than ls. YMMV.

  • 4
    The shell will sort the *. So this way is probably still very slow for 5 million files. – Mikel Mar 18 '14 at 13:52
  • 3
    @Mikel More than that, I'm pretty sure that 5 million files is over the point where globbing will break entirely. – evilsoup Mar 18 '14 at 15:26
  • 4
    Minimum file name length (for 5 million files) is 3 characters (maybe 4 if you stick to more common characters) plus delimiters = 4 chars per file, i.e 20 MB of command arguments. That is well over the common 2MB expanded command line length. Exec (and even the builtins) would baulk. – Johan Mar 19 '14 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.