524

When administering Linux systems I often find myself struggling to track down the culprit after a partition goes full. I normally use du / | sort -nr but on a large filesystem this takes a long time before any results are returned.

Also, this is usually successful in highlighting the worst offender but I've often found myself resorting to du without the sort in more subtle cases and then had to trawl through the output.

I'd prefer a command line solution which relies on standard Linux commands since I have to administer quite a few systems and installing new software is a hassle (especially when out of disk space!)

migrated from stackoverflow.com Apr 18 '14 at 18:36

This question came from our site for professional and enthusiast programmers.

  • 1
    @Bart thanks for taking the time to improve posts here, but may I ask you to take a little more care when adding tags? Tags aren’t for visibility, they describe the question. The disk tag isn’t appropriate here (see its tag description), and the at least two of the tags you added in this suggested edit weren’t appropriate there (Kali isn’t Debian, and there are no PPAs involved). – Stephen Kitt Jun 26 at 10:02

37 Answers 37

598

Try ncdu, an excellent command-line disk usage analyser:

  • 5
    Typically, I hate being asked to install something to solve a simple issue, but this is just great. – gwg Jul 5 '16 at 18:58
  • 12
    sudo apt install ncdu on ubuntu gets it easily. It's great – Orion Edwards Jul 19 '17 at 22:30
  • 10
    You quite probably know which filesystem is short of space. In which case you can use ncdu -x to only count files and directories on the same filesystem as the directory being scanned. – Luke Cousins Jul 21 '17 at 11:51
  • 6
    best answer. also: sudo ncdu -rx / should give a clean read on biggest dirs/files ONLY on root area drive. (-r = read-only, -x = stay on same filesystem (meaning: do not traverse other filesystem mounts) ) – bshea Sep 21 '17 at 15:52
  • 3
    I have so little space that I can't install ncdu – Chris Jun 14 '18 at 16:57
335

Don't go straight to du /. Use df to find the partition that's hurting you, and then try du commands.

One I like to try is

# U.S.
du -h <dir> | grep '[0-9\.]\+G'
# Others
du -h <dir> | grep '[0-9\,]\+G'

because it prints sizes in "human readable form". Unless you've got really small partitions, grepping for directories in the gigabytes is a pretty good filter for what you want. This will take you some time, but unless you have quotas set up, I think that's just the way it's going to be.

As @jchavannes points out in the comments, the expression can get more precise if you're finding too many false positives. I incorporated the suggestion, which does make it better, but there are still false positives, so there are just tradeoffs (simpler expr, worse results; more complex and longer expr, better results). If you have too many little directories showing up in your output, adjust your regex accordingly. For example,

grep '^\s*[0-9\.]\+G'

is even more accurate (no < 1GB directories will be listed).

If you do have quotas, you can use

quota -v

to find users that are hogging the disk.

  • 2
    This is very quick, simple and practical – zzapper Oct 29 '12 at 16:43
  • 24
    grep '[0-9]G' contained a lot of false positives and also omitted any decimals. This worked better for me: sudo du -h / | grep -P '^[0-9\.]+G' – jchavannes Aug 14 '14 at 6:09
  • 1
    In case you have really big directories, you'll want [GT] instead of just G – Vitruvius Mar 28 '15 at 20:20
  • 1
    Is there a tool that will continuously monitor disk usage across all directories (lazily) in the filesystem? Something that can be streamed to a web UI? Preferably soft-realtime information. – CMCDragonkai May 7 '15 at 9:00
  • 20
    I like to use du -h | sort -hr | head – augurar Jun 13 '16 at 18:48
104

For a first look, use the “summary” view of du:

du -s /*

The effect is to print the size of each of its arguments, i.e. every root folder in the case above.

Furthermore, both GNU du and BSD du can be depth-restricted (but POSIX du cannot!):

  • GNU (Linux, …):

    du --max-depth 3
    
  • BSD (macOS, …):

    du -d 3
    

This will limit the output display to depth 3. The calculated and displayed size is still the total of the full depth, of course. But despite this, restricting the display depth drastically speeds up the calculation.

Another helpful option is -h (words on both GNU and BSD but, once again, not on POSIX-only du) for “human-readable” output (i.e. using KiB, MiB etc.).

  • 23
    if du complains about -d try --max-depth 5 in stead. – ReactiveRaven Jul 2 '13 at 11:25
  • 8
    Great anwser. Seems correct for me. I suggest du -hcd 1 /directory. -h for human readable, c for total and d for depth. – Thales Ceolin Feb 4 '14 at 1:13
  • I'm use du -hd 1 <folder to inspect> | sort -hr | head – jonathanccalixto Jan 10 '17 at 19:39
  • du --max-depth 5 -h /* 2>&1 | grep '[0-9\.]\+G' | sort -hr | head to filter Permission denied – srghma Sep 1 '17 at 10:36
49

You can also run the following command using du :

~# du -Pshx /* 2>/dev/null
  • The -s option summarizes and displays total for each argument.
  • h prints Mio, Gio, etc.
  • x = stay in one filesystem (very useful).
  • P = don't follow symlinks (which could cause files to be counted twice for instance).

Be careful, the /root directory will not be shown, you have to run ~# du -Pshx /root 2>/dev/null to get that (once, I struggled a lot not pointing out that my /root directory had gone full).

Edit: Corrected option -P

  • 2
    du -Pshx .* * 2>/dev/null + hidden/system directories – Mykhaylo Adamovych Feb 15 '16 at 10:16
27

Finding the biggest files on the filesystem is always going to take a long time. By definition you have to traverse the whole filesystem looking for big files. The only solution is probably to run a cron job on all your systems to have the file ready ahead of time.

One other thing, the x option of du is useful to keep du from following mount points into other filesystems. I.e:

du -x [path]

The full command I usually run is:

sudo du -xm / | sort -rn > usage.txt

The -m means return results in megabytes, and sort -rn will sort the results largest number first. You can then open usage.txt in an editor, and the biggest folders (starting with /) will be at the top.

  • 3
    Thanks for pointing out the -x flag! – SamB Jun 2 '10 at 20:55
  • 1
    "finding biggest takes long time.." -> Well it depends, but tend to disagree: doesn't take that long with utilities like ncdu - at least quicker than du or find (depending on depth and arguments).. – bshea Sep 21 '17 at 15:35
  • since I prefer not to be root, I had to adapt where the file is written : sudo du -xm / | sort -rn > ~/usage.txt – Bruno Sep 14 '18 at 6:55
20

I always use du -sm * | sort -n, which gives you a sorted list of how much the subdirectories of the current working directory use up, in mebibytes.

You can also try Konqueror, which has a "size view" mode, which is similar to what WinDirStat does on Windows: it gives you a viual representation of which files/directories use up most of your space.

Update: on more recent versions, you can also use du -sh * | sort -h which will show human-readable filesizes and sort by those. (numbers will be suffixed with K, M, G, ...)

For people looking for an alternative to KDE3's Konqueror file size view may take a look at filelight, though it's not quite as nice.

  • That's only Konqueror 3.x though - the file size view still hasn't been ported to KDE4. – flussence Feb 9 '09 at 19:09
  • 'du -sh * | sort -h ' works perfectly on my Linux (Centos distro) box. Thanks! – pahariayogi Apr 25 '17 at 14:05
18

I use this for the top 25 worst offenders below the current directory

# -S to not include subdir size, sorted and limited to top 25
du -S . | sort -nr | head -25
  • This command did the trick to find a hidden folder that seemed to be increasing in size over time. Thanks! – thegreendroid Jun 20 '13 at 2:24
  • Is this in bytes? – User Sep 17 '14 at 0:12
  • By default, on my system, 'du -S' gives a nice human readable output. You get a plain number of bytes for small files, then a number with a 'KB' or 'MB' suffix for bigger files. – serg10 Sep 17 '14 at 8:48
  • You could do du -Sh to get a human readable output. – Siddhartha Jan 26 '16 at 2:57
  • @Siddhartha If you add -h, it will likely change the effect of the sort -nr command - meaning the sort will no longer work, and then the head command will also no longer work – Clare Macrae Dec 4 '17 at 13:00
14

At a previous company we used to have a cron job that was run overnight and identified any files over a certain size, e.g.

find / -size +10000k

You may want to be more selective about the directories that you are searching, and watch out for any remotely mounted drives which might go offline.

  • You can use the -x option of find to make sure you don't find files on other devices than the start point of your find command. This fixes the remotely mounted drives issue. – rjmunro Jun 29 '15 at 16:29
10

One option would be to run your du/sort command as a cron job, and output to a file, so it's already there when you need it.

9

For the commandline I think the du/sort method is the best. If you're not on a server you should take a look at Baobab - Disk usage analyzer. This program also takes some time to run, but you can easily find the sub directory deep, deep down where all the old Linux ISOs are.

  • 2
    It can also scan remote folders via SSH, FTP, SMB and WebDAV. – Colonel Sponsz Dec 2 '08 at 16:34
  • This is great. Some things just work better with a GUI to visualize them, and this is one of them! I need an X-server on my server anyways for CrashPlan, so it works on that too. – timelmer Jun 25 '16 at 20:46
9

I use

du -ch --max-depth=2 .

and I change the max-depth to suit my needs. The "c" option prints totals for the folders and the "h" option prints the sizes in K, M, or G as appropriate. As others have said, it still scans all the directories, but it limits the output in a way that I find easier to find the large directories.

9

I'm going to second xdiskusage. But I'm going to add in the note that it is actually a du frontend and can read the du output from a file. So you can run du -ax /home > ~/home-du on your server, scp the file back, and then analyze it graphically. Or pipe it through ssh.

6

Try feeding the output of du into a simple awk script that checks to see if the size of the directory is larger than some threshold, if so it prints it. You don't have to wait for the entire tree to be traversed before you start getting info (vs. many of the other answers).

For example, the following displays any directories that consume more than about 500 MB.

du -kx / | awk '{ if ($1 > 500000) { print $0} }'

To make the above a little more reusable, you can define a function in your .bashrc, ( or you could make it into a standalone script).

dubig() {
    [ -z "$1" ] && echo "usage: dubig sizethreshMB [dir]" && return
    du -kx $2 | awk '{ if ($1 > '$1'*1024) { print $0} }'
}

So dubig 200 ~/ looks under the home directory (without following symlinks off device) for directories that use more than 200 MB.

  • It's a pity that a dozen of grep hacks are more upvoted. Oh and du -k will make it absolutely certain that du is using KB units – ndemou Nov 23 '16 at 20:05
  • Good idea about the -k. Edited. – Mark Borgerding Nov 24 '16 at 11:16
  • Even simpler and more robust: du -kx $2 | awk '$1>'$(($1*1024)) (if you specify only a condition aka pattern to awk the default action is print $0) – dave_thompson_085 Nov 27 '16 at 11:31
  • Good point @date_thompson_085. That's true for all versions of awk I know of (net/free-BSD & GNU). @mark-borgerding so this means that you can greatly simplify your first example to just du -kx / | awk '$1 > 500000' – ndemou Dec 13 '16 at 9:46
  • @mark-borgerding: If you have just a few kBytes left somewhere you can also keep the whole output of du like this du -kx / | tee /tmp/du.log | awk '$1 > 500000'. This is very helpful because if your first filtering turns out to be fruitless you can try other values like this awk '$1 > 200000' /tmp/du.log or inspect the complete output like this sort -nr /tmp/du.log|less without re-scanning the whole filesystem – ndemou Dec 13 '16 at 9:59
4

I like the good old xdiskusage as a graphical alternative to du(1).

  • Note this part of the question: "I'd prefer a command line solution which relies on standard Linux commands since..." – ndemou Jul 4 '17 at 20:20
4

I prefer to use the following to get an overview and drill down from there...

cd /folder_to_check
du -shx */

This will display results with human readable output such as GB, MB. It will also prevent traversing through remote filesystems. The -s option only shows summary of each folder found so you can drill down further if interested in more details of a folder. Keep in mind that this solution will only show folders so you will want to omit the / after the asterisk if you want files too.

4

Not mentioned here but you should also check lsof in case of deleted/hanging files. I had a 5.9GB deleted tmp file from a run away cronjob.

https://serverfault.com/questions/207100/how-can-i-find-phantom-storage-usage Helped me out in find the process owner of said file ( cron ) and then I was able to goto /proc/{cron id}/fd/{file handle #} less the file in question to get the start of the run away, resolve that, and then echo "" > file to clear up space and let cron gracefully close itself up.

3

From the terminal, you can get a visual representation of disk usage with dutree

It is very fast and light because it is implemented in Rust

dutree

$ dutree -h
Usage: dutree [options] <path> [<path>..]

Options:
    -d, --depth [DEPTH] show directories up to depth N (def 1)
    -a, --aggr [N[KMG]] aggregate smaller than N B/KiB/MiB/GiB (def 1M)
    -s, --summary       equivalent to -da, or -d1 -a1M
    -u, --usage         report real disk usage instead of file size
    -b, --bytes         print sizes in bytes
    -f, --files-only    skip directories for a fast local overview
    -x, --exclude NAME  exclude matching files or directories
    -H, --no-hidden     exclude hidden files
    -A, --ascii         ASCII characters only, no colors
    -h, --help          show help
    -v, --version       print version number

See all the usage details in the website

2

For command line du (and it's options) seems to be the best way. DiskHog looks like it uses du/df info from a cron job too so Peter's suggestion is probably the best combination of simple and effective.

(FileLight and KDirStat are ideal for GUI.)

2

You can use standard tools like find and sort to analyze your disk space usage.

List directories sorted by their size:

find / -mount -type d -exec du -s "{}" \; | sort -n

List files sorted by their size:

find / -mount -printf "%k\t%p\n" | sort -n
  • 1
    I find this to be the best answer, to detect the large sized in sorted order – vimal krishna Oct 1 '17 at 11:16
2

Maybe worth to note that mc (Midnight Commander, a classic text-mode file manager) by default show only the size of the directory inodes (usually 4096) but with CtrlSpace or with menu Tools you can see the space occupied by the selected directory in a human readable format (e.g., some like 103151M).

For instance, the picture below show the full size of the vanilla TeX Live distributions of 2018 and 2017, while the versions of 2015 and 2016 show only the size of the inode (but they have really near of 5 Gb each).

That is, CtrlSpace must be done one for one, only for the actual directory level, but it is so fast and handy when you are navigating with mc that maybe you will not need ncdu (that indeed, only for this purpose is better). Otherwise, you can also run ncdu from mc. without exit from mc or launch another terminal.

mwe

1

At first I check the size of directories, like so:

du -sh /var/cache/*/
1

If you know that the large files have been added in the last few days (say, 3), then you can use a find command in conjunction with "ls -ltra" to discover those recently added files:

find /some/dir -type f -mtime -3 -exec ls -lart {} \;

This will give you just the files ("-type f"), not directories; just the files with modification time over the last 3 days ("-mtime -3") and execute "ls -lart" against each found file ("-exec" part).

1

To understand disproportionate disk space usage it's often useful to start at the root directory and walk up through some of its largest children.

We can do this by

  • saving the output of du into a file
  • grepping through the result iteratively

That is:

# sum up the size of all files and directories under the root filesystem
du -a -h -x / > disk_usage.txt
# display the size of root items
grep $'\t/[^/]*$' disk_usage.txt

now let's say /usr appear too large

# display the size of /usr items
grep $'\t/usr/[^/]*$' disk_usage.txt

now if /usr/local is suspiciously large

# display the size /usr/local items
grep $'\t/usr/local/[^/]*$' disk_usage.txt

and so on...

1

I have used this command to find files bigger than 100Mb:

find / -size +100M -exec ls -l {} \;
0

I've had success tracking down the worst offender(s) piping the du output in human readable form to egrep and matching to a regular expression.

For example:

du -h | egrep "[0-9]+G.*|[5-9][0-9][0-9]M.*"

which should give you back everything 500 megs or higher.

  • Don't use grep for arithmetic operations -- use awk instead: du -k | awk '$1 > 500000'. It is much easier to understand, edit and get correct on the first try. – ndemou Jul 4 '17 at 20:25
0

If you want speed, you can enable quotas on the filesystems you want to monitor (you need not set quotas for any user), and use a script that uses the quota command to list the disk space being used by each user. For instance:

quota -v $user | grep $filesystem | awk '{ print $2 }'

would give you the disk usage in blocks for the particular user on the particular filesystem. You should be able to check usages in a matter of seconds this way.

To enable quotas you will need to add usrquota to the filesystem options in your /etc/fstab file and then probably reboot so that quotacheck can be run on a idle filesystem before quotaon is called.

0

Here is a tiny app that uses deep sampling to find tumors in any disk or directory. It walks the directory tree twice, once to measure it, and the second time to print out the paths to 20 "random" bytes under the directory.

void walk(string sDir, int iPass, int64& n, int64& n1, int64 step){
    foreach(string sSubDir in sDir){
        walk(sDir + "/" + sSubDir, iPass, n, n1, step);
    }
    foreach(string sFile in sDir){
        string sPath = sDir + "/" + sFile;
        int64 len = File.Size(sPath);
        if (iPass == 2){
            while(n1 <= n+len){
               print sPath;
               n1 += step;
            }
        }
        n += len;
    }
}

void dscan(){
    int64 n = 0, n1 = 0, step = 0;
    // pass 1, measure
    walk(".", 1, n, n1);
    print n;
    // pass 2, print
    step = n/20; n1 = step/2; n = 0;
    walk(".", 2, n, n1);
    print n;
}

The output looks like this for my Program Files directory:

 7,908,634,694
.\ArcSoft\PhotoStudio 2000\Samples\3.jpg
.\Common Files\Java\Update\Base Images\j2re1.4.2-b28\core1.zip
.\Common Files\Wise Installation Wizard\WISDED53B0BB67C4244AE6AD6FD3C28D1EF_7_0_2_7.MSI
.\Insightful\splus62\java\jre\lib\jaws.jar
.\Intel\Compiler\Fortran\9.1\em64t\bin\tselect.exe
.\Intel\Download\IntelFortranProCompiler91\Compiler\Itanium\Data1.cab
.\Intel\MKL\8.0.1\em64t\bin\mkl_lapack32.dll
.\Java\jre1.6.0\bin\client\classes.jsa
.\Microsoft SQL Server\90\Setup Bootstrap\sqlsval.dll
.\Microsoft Visual Studio\DF98\DOC\TAPI.CHM
.\Microsoft Visual Studio .NET 2003\CompactFrameworkSDK\v1.0.5000\Windows CE\sqlce20sql2ksp1.exe
.\Microsoft Visual Studio .NET 2003\SDK\v1.1\Tool Developers Guide\docs\Partition II Metadata.doc
.\Microsoft Visual Studio .NET 2003\Visual Studio .NET Enterprise Architect 2003 - English\Logs\VSMsiLog0A34.txt
.\Microsoft Visual Studio 8\Microsoft Visual Studio 2005 Professional Edition - ENU\Logs\VSMsiLog1A9E.txt
.\Microsoft Visual Studio 8\SmartDevices\SDK\CompactFramework\2.0\v2.0\WindowsCE\wce500\mipsiv\NETCFv2.wce5.mipsiv.cab
.\Microsoft Visual Studio 8\VC\ce\atlmfc\lib\armv4i\UafxcW.lib
.\Microsoft Visual Studio 8\VC\ce\Dll\mipsii\mfc80ud.pdb
.\Movie Maker\MUI\0409\moviemk.chm
.\TheCompany\TheProduct\docs\TheProduct User's Guide.pdf
.\VNI\CTT6.0\help\StatV1.pdf
7,908,634,694

It tells me that the directory is 7.9gb, of which

  • ~15% goes to the Intel Fortran compiler
  • ~15% goes to VS .NET 2003
  • ~20% goes to VS 8

It is simple enough to ask if any of these can be unloaded.

It also tells about file types that are distributed across the file system, but taken together represent an opportunity for space saving:

  • ~15% roughly goes to .cab and .MSI files
  • ~10% roughly goes to logging text files

It shows plenty of other things in there also, that I could probably do without, like "SmartDevices" and "ce" support (~15%).

It does take linear time, but it doesn't have to be done often.

Examples of things it has found:

  • backup copies of DLLs in many saved code repositories, that don't really need to be saved
  • a backup copy of someone's hard drive on the server, under an obscure directory
  • voluminous temporary internet files
  • ancient doc and help files long past being needed
0

I had a similar issue, but the answers on this page weren't enough. I found the following command to be the most useful for the listing:

du -a / | sort -n -r | head -n 20

Which would show me the 20 biggest offenders. However even though I ran this, it did not show me the real issue, because I had already deleted the file. The catch was that there was a process still running that was referencing the deleted log file... so I had to kill that process first then the disk space showed up as free.

  • Good point but this should be a comment and not an answer by itself - this question suffers from too many answers – ndemou Jun 9 '17 at 10:36
0

You can use DiskReport.net to generate an online web report of all your disks.

With many runs it will show you history graph for all your folders, easy to find what has grow

  • This tool doesn't match two main points of the question "I often find myself struggling to track down the culprit after a partition goes full" and "I'd prefer a command line solution which relies on standard Linux commands" – ndemou Jun 9 '17 at 10:35
0

There is a nice piece of cross-platform freeware called JDiskReport which includes a GUI to explore what's taking up all that space.

Example screenshot:
JDiskReport screenshot

Of course, you'll need to clear up a little bit of space manually before you can download and install it, or download this to a different drive (like a USB thumbdrive).


(Copied here from same-author answer on duplicate question)