Fast way to display the size of each subdirectory in a directory

Question

I want to check which directories take most disk space, quickly.

I tried du -sh subdir but it took more than 20 seconds on bigger directories.

I'm not sure how to display the size of all the subdirectories in the home directory at once with this method, but I'm afraid that it might take minutes...

Is there a fast way to do this?

I don't need to display the size of files, just directories.

Answer 1

Sample directory

$ ls -aF
./  ../  .asd/  folder1/  folder2/  list  t1  t2  xyz/

To find sizes only for folders, excluding hidden folders:

$ find -type d -name '[!.]*' -exec du -sh {} + 
4.0K    ./folder1
4.0K    ./folder2
8.0K    ./xyz

If you need a total at the end as well:

$ find -type d -name '[!.]*' -exec du -ch {} + 
4.0K    ./folder1
4.0K    ./folder2
8.0K    ./xyz
16K total

To sort the results:

$ find -type d -name '[!.]*' -exec du -sh {} + | sort -h
4.0K    ./folder1
4.0K    ./folder2
8.0K    ./xyz

To reverse the sorting order:

$ find -type d -name '[!.]*' -exec du -sh {} + | sort -hr
8.0K    ./xyz
4.0K    ./folder2
4.0K    ./folder1

If you need with hidden directories as well, remove -name '[!.]*' from find command. I don't know any other command to find size of folders that is faster than du. Use df for file system disk space usage

Use find -maxdepth 1 -type d -name '[!.]*' -exec du -sh {} + to avoid sub-folders showing up

Answer 2

Use package ncdu. you can install it by sudo apt-get install ncdu.
on server you can use it with options ncdu -q -x (Quiet Mode and Omit mounted directories).

Ncdu vs du/df

The interface of ncdu is built using ncurses and is interactive. Ncdu is different from df or du. Ncdu just does the one task of reporting the space used by a directory and drill down. On the other hand the df command reports space used by different storage devices.

So ncdu is a powerful tool to monitor, check and analyse disk space usage on your linux system

Answer 3

All folders occupy the same amount of space, namely 4096 bytes. You don't want to know the size of the folders, but the size of what's in them. And this demands counting, which in turn demands time.

du counts directory sizes by default. So to get the sizes for it either:

cd && du

Or

du ~

See man du for more options.

Answer 4

Use the following command, it will display quickly the top 10 directories according to the size occupied in the system:

du -hsx /* | sort -rh | head -10

Output e.g.

[root@x ~]# du -hxs /* | sort -rh | head -10

10G     /mnt
5.4G    /usr
1.5G    /var
418M    /lib
274M    /opt
224M    /root
55M     /boot
36M     /home
30M     /lib64
16M     /sbin

Answer 5

As others have said, it is just a slow process. We have multi user machines and the worst is when a disk runs low and several people start du at the same time. So, we use a cached view of the disk usage (we run it twice a day). There are lots out there, but my faviourite from a visual perspective is duc

http://duc.zevv.nl/

Indexes are generated from cron jobs like below - this one specifies 6 levels of hierarchy in the index

30 5,12 * * * /usr/local/bin/duc index -x -m 6 -d /mydisk/ducdb_1.4.sqlite /mydisk/users >/dev/null 2>&1

Then we use the Web front end (duc cgi) to view the index. It is very slick and aesthetically great. Many thanks to the authors Ico Doornekamp & John Stoffel

Answer 6

You can list the top 10 folders by size using this command below.

find $(pwd) -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

Answer 7

If running du on a particular directory takes 20 seconds, then that is what it will take to run du on that directory. You can't really do it quicker.

What you can do quicker is to eliminate the manual labor, i.e. remove the need to run du manually on each individual directory and to compare the sizes.

You can do this in a few different ways, depending on how you want to count the sizes. If your subdirectories of your home directory contains hard links (files that have multiple names in the file hierarchy), and you want to count these once per name, you can do this with the -l ("ell", not "one") option to GNU du. With standard du, multiple hard links would be counted separately only if encountered in separate du runs.

You also have to decide whether you want to count the apparent ("logical") size, i.e. the number of bytes that a program reading the file from start to finish would read, or the disk usage ("physical") size, i.e. the number of bytes actually used for storing the file on disk. These two sizes may be different from each other if a file is sparse, i.e. if it contains holes. Unallocated disk images or swap files are typically sparse. GNU du can give you the "logical" sizes using --apparent-size or --bytes, but standard du can't.

Assuming that you are only interested in actual disk usage of each directory, individually, i.e. that you want to count hard links multiple times only if they are located in different subdirectories under $HOME, and that you want to count the actual, "physical", size of any sparse files, then you may loop over the directories in your home directory and run du on each one like so:

for dirpath in "$HOME"/*/; do
    du -k -s -- "$dirpath"
done

The pattern "$HOME"/*/ will expand to a list of pathnames of (visible) directories beneath your home directory, and/or to visible symbolic links to these.

This will give you the sizes in kilobytes. Many du implementations can show you the "human readable", approximate, sizes if you use -h in place of -k (see man du).

You may sort the output of this loop numerically, with the largest directory appearing first, like so:

for dirpath in "$HOME"/*/; do
    du -k -s -- "$dirpath"
done | sort -n -r

If you used -h in place of -k with du, you will need to have a sort that understands "human readable" units. Some sort implementations do this with their -h option (see man sort).

Answer 8

You may try this: This solution assumes a lots:

• that your ultimate goal is to quickly determine the biggest directory (NOT counting its subdirs) to for exemple find quickly where you should do some cleaning up to free space (an often needed thing)
• that the filenames do not contain spaces/tabs/newlines/etc
• that the size of a file is always in the 7th field when outputted by find ... -ls :

And it is restricted to using basic versions of : find, awk (not the GNU versions, which would help by proposing better ways to display the filename and its size)

find ./ -type f -ls | awk ' 
    { pathname=$0; sub("[^/]*$","",pathname); size[pathname]+=$7 } 
END { for(dirname in size){print size[dirname],dirname }
    }' | sort -n

This will output directly the biggest directories at the end, and each size displayed is only "this directories content, not counting subdirs", which will help you to aim directly at the biggest occupancies. (du also adding subdirs can say "A/" is the biggest but it is in fact "A/B/C" (or even anotherdir/E/F/) that is the biggest without counting any subdirs...

Of course in the real world: your files may contain spaces, etc, and this will require lots of tweaking. One possibility is then to instead use GNU find, as with it you can only output: size_of_file name of file here, and thus add field $1, and the rest of the line is used to compute the dirpath. But there will always be special cases not covered [until someone else proposes a better version]

Answer 9

A simple command is ls | xargs du -sh