0

Summary

What command can I run in a bash script that will make the next ls at a terminal list the files in a directory immediately, instead of having to wait 10 seconds?

And see Questions 1 and 2 at end of "Full Details".

Full Details

I've got 700 video files (mp4) in a directory, ranging in size from 100 MB to 1GB. The average file size is 200 MB.

When I list them for the first time after boot (using ls in a terminal, or using pcmanfm), it takes 10 seconds before the files are listed. Thereafter, when I list them again (with ls or in pcmanfm), the files are listed almost immediately.

So to try and get round this initial wait time, I ran the following command in a script after bootup:

ls /path/to/vid-directory

... so that thereafter, when I list the directory with ls or in pcmanfm, the files list immediately.

But strangely, running

ls /path/to/vid-directory

in the SCRIPT does NOT cause the NEXT ls or pcmanfm to list the files immediately (they're listed after 10 seconds). Also strangely, when the initial script runs, the files are listed almost immediately.

So, running ls at a terminal seems to store the filenames in RAM cache, whereas running ls in a script does not*.

Question 1: Why is the latter* happening?

Question 2: What command(s) should I run in my script that WILL cause the next ls or pcmanfm to list the files immediately?

PS: Whilst doing the testing, I made sure that RAM was initially empty by running

sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
Improve this question
5
  • 1
    What is the actual size of the directory itself? Did it once contain a huge number of files? After I flush caches, I can ls -lR about 1000 files a second.
    – Paul_Pedant
    Commented Jul 31, 2020 at 20:33
  • Can you please add some information about your operating system, the media your data is on (HDD/SSD? USB drive? Network file system?) and the file system you are using? Similarly to what Paul_Pedant says, I see ls listing ~1000 files/second from a USB 3 spinning HDD (~6000 files, ~500GB in total).
    – fra-san
    Commented Jul 31, 2020 at 20:59
  • @Paul_Pedant: Part 1 of 2: The directory with the 700 vids in is 135 GB in size. There are also 13 subdirectories (not nested), so the TOTAL size (including the just-mentioned directory) is 212 GB. My thinking is that the content of those subdirectories is not involved in the time it's taking ls to do the listing. The "ls /path/to/vid-directory" lists the files in the first directory, plus just the names of the subdirectories.
    – dave99
    Commented Aug 1, 2020 at 17:42
  • @Paul_Pedant: Part 2 of 2: I wouldn't say the first directory (the one with 700 files in) once contained a huge number of files (nor the subdirectories). My guesstimate is that it might have once contained 1100 files (vids). My instinct tells me that the behaviour isn't to do with some kind of "fault", but I could be wrong.
    – dave99
    Commented Aug 1, 2020 at 17:43
  • The actual file data size would never affect ls. For a short listing, it scans the entire directory (that is, just the list of names and inode numbers) once. It sorts it, so it can't write the first name until it reads the last. For a long listing, it also reads the inode, which is one block per file, and contains dates, sizes, permissions and timestamps. I'm wondering whether your PATH is overloaded -- maybe an early directory in there has a huge number of files. Bash hashes command names to avoid multiple searches: that would fit with different bash processes having an initial delay.
    – Paul_Pedant
    Commented Aug 1, 2020 at 18:13

2 Answers 2

Reset to default
3

The only plausible difference between ls in a terminal and ls in a script is that you've aliased ls to pass an option that requires it not only to read the list of file names, but also to read the file's attributes. The option is probably --color. Reading the list of file names only requires reading from the directory itself, whereas reading the file's attributes requires accessing each individual file's inode (with an lstat system call). So when you run ls in that script, only the file names end up in the cache, not their attributes. The first time you run ls in the terminal or use a GUI file manager, the attributes need to be loaded.

Pass the same options to ls in your script that you use on the terminal.

10 seconds to list only 700 files is abnormally slow though. The size of the files is irrelevant. Typical modern systems only start getting noticeably slow with tens or hundreds of thousands of files if not more.

Improve this answer
0
2

You can find out what the problem is by running after a reboot:

strace -tt -o ls.strace ls /path/to/directory

The output of ls can be accelerated by ls -U but, of course, that does change the output.

700 files is not a large number. Some acceleration may be achieved by rewriting all the directory entries:

mkdir /tmpdir/on/the/same/filesystem
mv * /tmpdir/on/the/same/filesystem
mv /tmpdir/on/the/same/filesystem/* .

But the most effective solution may be to have the directory be read automatically in the background at boot.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .