Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I had a directory which had around 5 million files. When I tried to run the ls command from inside this directory, my system consumed a huge amount of memory and it hung after sometime. Is there an efficient way to list the files other than using the ls command?

share|improve this question
5  
Make sure you don't have an alias for ls that uses --color or -F as that would mean doing a lstat(2) for each file. –  Stéphane Chazelas Mar 17 at 21:37
4  
By the way, storing millions of files in a single directory is a rather bad idea. If you control the directory layout, perhaps split it by some criteria? –  d33tah Mar 18 at 1:21
    
Was that a pure ls call or did you use options? –  Hauke Laging Mar 18 at 3:27
1  
@d33tah Yeah, 5 million is a lot! My root file system has a limit of 7 million inodes. –  Mikel Mar 18 at 14:05
4  
5 million items to output -how are you looking at this - simple listing is too much to see - so what do you want the listing for? –  Mark Mar 18 at 15:43
show 2 more comments

4 Answers 4

up vote 24 down vote accepted

ls actually sorts the files and tries to list them which becomes a huge overhead if we are trying to list more than a million files inside a directory. As mentioned in this link, we can use strace or find to list the files. However, those options also seemed unfeasible to my problem since I had 5 million files. After some bit of googling, I found that if we list the directories using getdents(), it is supposed to be faster, because ls, find and Python libraries use readdir() which is slower but uses getdents() underneath.

We can find the C code to list the files using getdents() from here:

/*
 * List directories using getdents() because ls, find and Python libraries
 * use readdir() which is slower (but uses getdents() underneath.
 *
 * Compile with 
 * ]$ gcc  getdents.c -o getdents
 */
#define _GNU_SOURCE
#include <dirent.h>     /* Defines DT_* constants */
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/syscall.h>

#define handle_error(msg) \
       do { perror(msg); exit(EXIT_FAILURE); } while (0)

struct linux_dirent {
   long           d_ino;
   off_t          d_off;
   unsigned short d_reclen;
   char           d_name[];
};

#define BUF_SIZE 1024*1024*5

int
main(int argc, char *argv[])
{
   int fd, nread;
   char buf[BUF_SIZE];
   struct linux_dirent *d;
   int bpos;
   char d_type;

   fd = open(argc > 1 ? argv[1] : ".", O_RDONLY | O_DIRECTORY);
   if (fd == -1)
       handle_error("open");

   for ( ; ; ) {
       nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
       if (nread == -1)
           handle_error("getdents");

       if (nread == 0)
           break;

       for (bpos = 0; bpos < nread;) {
           d = (struct linux_dirent *) (buf + bpos);
           d_type = *(buf + bpos + d->d_reclen - 1);
           if( d->d_ino != 0 && d_type == DT_REG ) {
              printf("%s\n", (char *)d->d_name );
           }
           bpos += d->d_reclen;
       }
   }

   exit(EXIT_SUCCESS);
}

Copy the C program above into directory in which the files need to be listed. Then execute the below commands.

gcc  getdents.c -o getdents
./getdents
share|improve this answer
10  
Could you add a small benchmark in timing for which your case does display with ls? –  Bernhard Mar 17 at 16:34
    
Sweet. And you could add an option to simply count the entries (files) rather than listing their names (saving millions of calls to printf, for this listing). –  ChuckCottrill Mar 17 at 20:50
11  
You know your directory is too big when you have to write custom code to list its contents... –  casey Mar 18 at 1:45
1  
@casey Except you don't have to. All this talk about getdents vs readdir misses the point. –  Mikel Mar 18 at 13:35
3  
Come on! It's already got 5 million files in there. Put your custom "ls" program into some other directory. –  Johan Mar 19 at 8:35
show 1 more comment

Avoid sorting by

ls -f
ls -U
ls --sort=none
share|improve this answer
2  
I wonder how much overhead the column layout adds, too. Adding the -1 flag could help. –  Mikel Mar 18 at 14:03
    
Probably not much, but every little bit helps, right? :) –  Mikel Mar 18 at 14:20
    
@Mikel Is that just a guess, or have you measured that? To me it seems that -1 takes even longer. –  Hauke Laging Mar 18 at 19:32
add comment

The most likely reason why it is slow is file type colouring, you can avoid this with \ls or /bin/ls turning off the colour options.

If you really have so many files in a dir, using find instead is also a good option.

share|improve this answer
2  
I don't think this should have been downvoted. Sorting is one problem, but even without sorting, ls -U --color would take a long time since it would stat each file. So both are correct. –  Mikel Mar 18 at 13:59
add comment

I find that echo * works much faster than ls. YMMV.

share|improve this answer
3  
The shell will sort the *. So this way is probably still very slow for 5 million files. –  Mikel Mar 18 at 13:52
1  
@Mikel More than that, I'm pretty sure that 5 million files is over the point where globbing will break entirely. –  evilsoup Mar 18 at 15:26
3  
Minimum file name length (for 5 million files) is 3 characters (maybe 4 if you stick to more common characters) plus delimiters = 4 chars per file, i.e 20 MB of command arguments. That is well over the common 2MB expanded command line length. Exec (and even the builtins) would baulk. –  Johan Mar 19 at 8:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.