2

I have almost 7,000 files in a single directory which makes loading that particular directory a tedious and time-consuming one. Is there any way to list and perform operations on files in that directory faster rather than distributing the files into multiple sub-folders(which makes it even more difficult)?

I'm using ext4. But also received the same from efs filesystems.

  • 1
    It's better to you make filter, you have some way, find, globbing and so on , But at first you explain exactly your task. – PersianGulf Mar 7 '15 at 20:22
  • 1
    Could you clarify what you mean by "loading"? Simply ls, or displaying the contents in a file manager (which one?), or something else? On my system listing 7,000 files with ls on an ext4 filesystem is pretty much instantaneous, as is displaying the directory in Thunar. – Stephen Kitt Mar 7 '15 at 20:56
  • @SnazzySanoj please answer comments, by amending your question. – ctrl-alt-delor Mar 7 '15 at 22:30
  • 1
    A lot of systems alias ls to something that makes it lstat every file. --color and -F do this. If you don't need ls to decorate the output, remove these aliases. Listing a 7000 file directory shouldn't be slow. – Mark Plotnick Mar 7 '15 at 23:48
0

A large directory can be problematic for a number of reasons other than simply listing of files. For one thing, the time it takes to open a file in that directory will increase because the directory has to be read until the file is found. On many filesystems including ext*, directory entries not organized nor optimized for retrieval efficiency.

Answering your specific question, I think that you'll find using ls takes awhile due to the sorting involved. A solution is to run ls unsorted (if there is such an option in your distribution of ls). Specifically, I might ls unsorted to a file and the sort it. I then have the file to refer to without having to do another ls for a bit.

Another similar method is to use the find command and listout the directory contents into a file (which will be unsorted) and then go from there.

Thus my suggestion would be based on accessibility, effeciency and easily finding files, to use multiple sub-directories.

  • "A directory is not organized to speed up retrievals of specific entries": that completely depends on the filesystem in use, and many commonly-used filesystem types do optimize directory indices for just this ability. – Celada Mar 8 '15 at 5:09
  • OK, I qualified my response. Could you provide the commonly used filesystems that do what you state for the record? – mdpc Mar 8 '15 at 5:42
  • 1
    XFS automatically, ext4 (probably ext3 as well?) with the "dir_index" option enabled (which is part of the default options) (see mkfs.ext4 manpage), and I didn't bother to look up references but probably most of the other modern filesystem types as well: Btrfs, ZFS, JFS, all support efficient indexing for large directories. – Celada Mar 8 '15 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.