4

I'm trying to find the top 10 big files from a directory which has lot of sub directories and N number of files. My find command is running forever as its listing all the files and then gets in to sort mode. I need a faster execution. In my latest run the listing itself wasn't finished in 15 mins and I had to terminate the execution.

I have used the following find commands, none of them is executing fast and they hang:

find /opt/app -type f -print 2>/dev/null | xargs ls -lS 2>/dev/null |
    sort -k5,5rn | head -10

find /opt/app -xdev -ls | awk '{print $7, $11}' | " sort -rn "}' | head

find /opt/app -type f -exec ls -lS {} + 2>/dev/null | head -n 10 |
    awk '{ print $5, $9 }'

What would be a better way to do this?

  • 2
    How many files in total? It could be that sorting the whole list is just slow. Since you only need the top ten, you could get a whole lot faster by storing and sorting only the ten largest. But I can't think of a tool that would do that automatically. It shouldn't be too hard to implement that in Perl or some other sufficiently high-level language (but I might just be promising too much). – ilkkachu May 16 '18 at 8:16
  • With zsh: ls -ldS /opt/app/**/*(DOL[1,10]) – Stéphane Chazelas May 16 '18 at 8:25
  • For an interactive method, you can use ncdu. It can sort by size. – Mioriin May 16 '18 at 8:32
  • yes its the listing which kills the time and then followed by sorting.. :( – Anoop Kumar KR May 16 '18 at 9:27
  • Since du -s * is very fast, you can implement a top-down algorithm that stops looking when a directory is smaller than the top-10 directories/files. If you have a few big files and many small files, this will be a lot faster then listing all files. – Raphael May 16 '18 at 9:34
5

In any case, you will need to read the content of all directories in there and do a lstat(), which implies reading inode data to get the file size for all the files before you can determine which file is the largest.

And that disk (or network for networked file systems) I/O access, unless all the information is already cached in memory is the part that will take the most time.

There are quite a few unnecessary things that your approach does like converting user/group ids to user/group names, compute calendar representation of time stamps, decode bytes into characters that is going to keep your CPUs busy in addition to the disks, but disk I/O is likely to still be the bottleneck.

You can skip most of that unnecessary processing if you use for instance zsh's globs and glob qualifiers:

ls -ldS /opt/apps/**/*(.DOL[1,10])

Or with (recent) GNU tools:

(export LC_ALL=C # avoid decoding bytes into characters and
                 # complex sorting orders. Note that it may affect
                 # the ls -l output format though.
 find /opt/apps -type f -printf '%s/%p\0' |
   sort -srnz |
   head -z |
   cut -zd/ -f2- |
   xargs -r0 ls -ldS)

And see if that makes it any quicker.

2

sort doesn't provide output until all items have been sorted. It can't go back in the output and new items, so that's how it is. In my tests, GNU sort becomes active as soon is it gets enough output to start on its sorting algorithms, but holds off on output until everything has been sorted, so both sort and find were be active during the same time period (though find was active earlier, and sort remained active later, there was overlap).

The fastest way I can think of using find is to avoid other commands and text processing as much as possible. Thus, print the size and name using find itself instead of running ls multiple times. Using GNU find (and assuming filenames without newlines):

find /opt/app -xdev -type f -printf "%s %p\n" | sort -k1,1n | head

Or you can run multiple finds in parallel:

(for d in /opt/app/*/; do find "$d" -xdev -type f -printf "%s %p\n" & done; wait) | sort -k1,1n | head

(this could turn out to be harmful because of output interleaving)

But you won't get any output until sort has done its work.

  • 1
    Those multiple finds will likely have their output improperly interleaved. It's unlikely to have any positive performance impact as the performance is going to be I/O bound here. Note that you don't need the wait as sort will wait for them anyway by virtue of waiting for eof on the pipe at the other end of which all the find's are writing. – Stéphane Chazelas May 16 '18 at 8:30
1

The following code will give to the top big files.

ls -l -R <folder path> | grep -v "^d" | awk '{print $5" "$9}' | sort -nrk1,1 | head -n10 | column -t

if you want the full path of the file as well, try the below:

find <folder path> -type f -exec du -a {} + | sort -nr | head -10
  • Thanks for the response Siva. The issue i face is /opt/app has 16 subdirectories then the 16 has other directories plus lots and lots of files. I have tried the commands you replied with, however its still executing for the last 10 mins and no response yet. This execution time is my biggest issue. – Anoop Kumar KR May 16 '18 at 8:01
  • i tried the first command and i got the output. but it took 14mins PRICE_HIST.ds.dsadm.dstageservsvc.0000.0000.0000.1ec00d2.5afb3193.0000.cd8f907c 7308574720 ITEM_LOC.ds.dsadm.dstageservsvc.0000.0002.0000.21300ee.5afb7964.0002.9d84a981 real 14m51.66s user 1m51.14s sys 0m39.59s – Anoop Kumar KR May 16 '18 at 9:26
  • Try the second command, might give a better response since we check only for files using -type – msp9011 May 16 '18 at 9:29
1

not sure if that what you need, but:

ls -lR $(pwd) | awk '
/:$/&&f{s=$0;f=0}
/:$/&&!f{sub(/:$/,"");s=$0;f=9;next}
NF&&f{ print s"/ "$0 }' | egrep -v '^.|^$|^total\ ' | awk '{print $6,$1$NF}' | sort -nk1 | tail -n 10
Output:
13307 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/library/dellemc_configure_raid.py
14029 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/library/dellemc_configure_idrac_eventing.py
15199 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/library/dellemc_configure_idrac_users.py
20294 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/library/dellemc_configure_idrac_network.py
20851 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/samples/dellemc_get_firmware_inventory.md
35149 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/COPYING.md
49569 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/docs/OMAM_1.0.1_Readme.pdf
65486 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/samples/dellemc_get_system_inventory.md
89526 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/docs/OMAM_1.0.1_Installation_Guide.pdf
438419 /home/pierdolia/repo/Dell-EMC-Ansible-Modules-for-iDRAC/docs/OMAM_1.0.1_Users_Guide.pdf
only one cons: if directory contains too many files, that ls could not handle, so this won't work.

in my personal exp, I have noticed, that find works longer than these little commands, yes it's not "final" version, can be shrunk, but it's quite good.

P.S. If you remove first AWK it will display only filenames:

ls -lR $(pwd) | egrep -v '^.|^$|^total\ ' | sort -nk5 | tail -n 10
-rw-rw-r-- 1 pierdolia pierdolia 13307 May 14 13:22 dellemc_configure_raid.py
-rw-rw-r-- 1 pierdolia pierdolia 14029 May 14 13:22 dellemc_configure_idrac_eventing.py
-rw-rw-r-- 1 pierdolia pierdolia 15199 May 14 13:22 dellemc_configure_idrac_users.py
-rw-rw-r-- 1 pierdolia pierdolia 20294 May 14 13:22 dellemc_configure_idrac_network.py
-rw-rw-r-- 1 pierdolia pierdolia 20851 May 14 13:22 dellemc_get_firmware_inventory.md
-rw-rw-r-- 1 pierdolia pierdolia 35149 May 14 13:22 COPYING.md
-rw-rw-r-- 1 pierdolia pierdolia  49569 May 14 13:22 OMAM_1.0.1_Readme.pdf
-rw-rw-r-- 1 pierdolia pierdolia 65486 May 14 13:22 dellemc_get_system_inventory.md
-rw-rw-r-- 1 pierdolia pierdolia  89526 May 14 13:22 OMAM_1.0.1_Installation_Guide.pdf
-rw-rw-r-- 1 pierdolia pierdolia 438419 May 14 13:22 OMAM_1.0.1_Users_Guide.pdf
or add additional awk:
ls -lR $(pwd) | egrep -v '^.|^$|^total\ ' | sort -nk5 | tail -n 10 | awk '{print $5,$NF}'
13307 dellemc_configure_raid.py
14029 dellemc_configure_idrac_eventing.py
15199 dellemc_configure_idrac_users.py
20294 dellemc_configure_idrac_network.py
20851 dellemc_get_firmware_inventory.md
35149 COPYING.md
49569 OMAM_1.0.1_Readme.pdf
65486 dellemc_get_system_inventory.md
89526 OMAM_1.0.1_Installation_Guide.pdf
438419 OMAM_1.0.1_Users_Guide.pdf 

  • I have ran the first command, unfortunately took 16 mins... ls -lR /opt/app | grep -v '^.|^$|^total\ ' | sort -k5,5rn | head -10 real 16m37.29s user 1m45.15s sys 0m43.71s – Anoop Kumar KR May 16 '18 at 11:48
  • how many files you have? – BiG_NoBoDy May 16 '18 at 13:57
  • there are 3393854 files – Anoop Kumar KR May 17 '18 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.