Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

We have a large file system on which a full du (disk usage) summary takes over two minutes. I'd like to find a way to speed up a disk usage summary for arbitrary directories on that file system.

For small branches I've noticed that du results seem to be cached somehow, as repeat requests are much faster, but on large branches the speed up become negligible.

Is there a simple way of speeding up du, or more aggressively caching results for branches that haven't been modified since the previous search?

Or is there an alternative command that can deliver disk usage summaries faster?

share|improve this question
3  
Two minutes doesn't seem that long to me. But the real question is: "Do you really want du to cache anything?" Shouldn't du give you exact, as-current-as-possible, real disk block counts? –  Bruce Ediger Mar 2 '11 at 17:47
    
I agree that replacing du would be bad, but a faster wrapper script with an identical interface would be very useful for us. Further, I would expect that caching results dependent on last-modified time (and assuming no disk-wide operations, eg. defragmentation) would give exact size results: am I missing something? –  Ian Mackinnon Mar 2 '11 at 18:11
1  
If you are concerned about too much disk usage you might consider implementing a quota. –  pyasi Mar 2 '11 at 19:20
    
Bruce - you could ask the same question about find. But then there's locate. –  Yuval Jul 19 '13 at 12:16
add comment

5 Answers

up vote 8 down vote accepted

What you are seeing when you rerun a du command is the effect of disk buffering. Once you read a block its disk buffer is kept in the buffer cache until that block is needed. For du you need to read the directory and the inode for each file in the directory. The du results are not cached in this case, but can be derived with far less disk IO.

While it would be possible to force the system to cache this information, overall performance would suffer as the required buffer space would not be available for actively accessed files.

The directory itself has no idea how large a file is, so each file's inode needs to be accessed. To keep the cached value up to date every time a file changed size the cached value would need to be updated. As a file can be listed in 0 or more directories this would require each file's inode to know which directories it is listed in. This would greatly complicate the inode structure and reduce IO performance. Also as du allows you to get results assuming different block sizes, the data required in the cache would need to increment or decrement the cached value for each possible block size further slowing performance.

share|improve this answer
add comment

As mentioned by SHW, agedu indeed created an index. I thought I'd share another way to create an index, after reading about locatedb. You can create your own version of a locatedb from du output:

du | awk '{print $2,$1}' | /usr/lib/locate/frcode > du.locatedb

awk rearranges the du output to have filenames first, so that frcode works right. Then use locate with this database to quickly report disk usage:

locate --database=du.locatedb pingus

You can expand this to suit your needs. I think it's a nice use of locatedb.

share|improve this answer
add comment

I have a cronjob set up to run updatedb every 10 mins. Keeps all the filesystem buffers nice and fresh. Might as well use that cheap RAM for something good. Use slabtop see 'before' and 'after'.

share|improve this answer
    
I don't understand how your answer relates to the question. updatedb says nothing about disk usage. If you're doing it just to traverse the disk, you're going to hurt overall performance. –  Gilles May 5 '11 at 21:54
1  
Counting up file sizes for du is slow because you have to access metadata of a potentially large number of files, scattered around the disk. If you run updatedb aggressively, the metadata for all the files is forced to be stored in RAM. The next time you run any other metadata-heavy operation, instead of doing thousands of seeks across the disks, you use the cache. Normally you have a small chance of having that particular portion of the tree's metadata cached. With my 'metadata cache priming' it's highly probable that the data you want is freshly cached. No physical seeks == FAST. –  Marcin May 6 '11 at 10:33
add comment

I prefer to use the agedu

share|improve this answer
2  
Doesn't answer the question, but still +1. Nice tip. –  0xC0000022L May 6 '11 at 23:15
add comment

If you can arrange for the different hierarchies of files to belong to different groups, you can set up disk quotas. Don't give an upper limit (or make it the size of the disk) unless you want one. You'll still be able to tell instantly how much of its (effectively infinite) quota the group is using.

This does require that your filesystem supports per-group quotas. Linux's Ext[234] and Solaris/*BSD/Linux's zfs do. It would be nice for your use case if group quotas took ACLs into account, but I don't think they do.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.