3

I have a program that is downloading some images into a folder. I want to count the number of images in the folder currently. I used a command like ls -1 download_folder | wc -l.

But this command takes a lot of time to run(since I have already downloaded a lot of images). While this command is running, my downloader keeps on downloading the images.

And finally, my command returns the count. I wanted to know, from what timestamp this count is. Is it from the instant I started running the command or it gives the current latest count?

Edit: I tried validating for myself, but could not find any way. Thanks.

2
  • I wonder if set -- *; echo $# would be faster or slower. It still sorts, but only uses shell built-ins.
    – Quasímodo
    Jan 29, 2021 at 11:16
  • @Quasímodo I assume "a lot of images" is enough to slow ls -1 significantly. I tried 100,000 files. ls takes 0m1.397s; ls -U takes 0m0.209s; set -- *; echo $# takes 0m8.933s. Jan 29, 2021 at 21:21

1 Answer 1

1

The ls command is reading the whole directory, and sorting it, and then passing it to wc, which does not care about order anyway. It will run much faster if you tell ls not to sort, as:

ls -U | wc -l

This will reduce the time window in which new files can be added, and be more accurate. However, there is still a race condition between the additions and the listing, which cannot be overcome. Some file systems hash the names, some append to a linear list, so ls may or may not see any additions while it runs.

2
  • where can I read about how my file system handles these cases? Jan 29, 2021 at 11:35
  • @satindersingh Each file system type has its specification which includes strengths and weaknesses. I start with Wikipedia "File system" and have ext4 and NTFS partitions. In the absence of explicit locking strategies, I always assume a file system will interleave system calls from different processes in the most inconvenient way. As in quantum mechanics, "anything that is not specifically forbidden is guaranteed to happen eventually". Jan 29, 2021 at 21:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .