5

Jenkins has stopped building and I can see from df: that /var is maxed. Following this I can see the culprit is /var/lib/jenkins

52K     alternatives
...
4.0K    games
2.3G    jenkins   <--this one
4.0K    logrotate.status
...
12M     yum
2.4G    total

However, when I enter the jenkins dir to see which folder is consuming most of this 2.3GB, there are some strange results:

$ sudo du -csh *
16K     config.xml
0       Connection Activity monitoring to slaves.log
0       Download metadata.log
0       Fingerprint cleanup.log
7.3M    fingerprints
4.0K    hudson.maven.MavenModuleSet.xml
4.0K    hudson.model.UpdateCenter.xml
4.0K    hudson.scm.CVSSCM.xml
4.0K    hudson.scm.SubversionSCM.xml
4.0K    hudson.tasks.Ant.xml
4.0K    hudson.tasks.Mailer.xml
4.0K    hudson.tasks.Maven.xml
4.0K    hudson.tasks.Shell.xml
4.0K    hudson.triggers.SCMTrigger.xml
4.0K    identity.key.enc
4.0K    jenkins.diagnostics.ooom.OutOfOrderBuildMonitor
4.0K    jenkins.model.ArtifactManagerConfiguration.xml
4.0K    jenkins.model.DownloadSettings.xml
4.0K    jenkins.model.JenkinsLocationConfiguration.xml
4.0K    jenkins.mvn.GlobalMavenConfig.xml
4.0K    jenkins.security.QueueItemAuthenticatorConfiguration.xml
160M    jobs
4.0K    nodeMonitors.xml
12K     Out of order build detection.log
41M     plugins
4.0K    proxy.xml
4.0K    queue.xml.bak
4.0K    secret.key
0       secret.key.not-so-secret
32K     secrets
1.1M    updates
8.0K    userContent
12K     users
4.0K    Workspace clean-up.log
209M    total

From 2.3GB to 209M. Can anyone explain this so I can create some space? I've deleted the Jenkins workspaces from the web frontend but still see this result.

1
  • 1
    I'm seeing that du -ch (instead of du -csh) is returning more results I need. If anyone can explain this I can accept that as an answer.
    – TenLeftFingers
    Nov 5, 2014 at 12:04

4 Answers 4

Reset to default
8

With this command,

sudo du -csh *

you are missing hidden directories, i.e. * expands to all names starting with anything but a dot (.). That means all directory names starting with a dot are not passed to the du command and their size is not taken into account.

In most situations, adding .??* to the parameters would fit the needs :

sudo du -csh .??* *

The extra .??* is expanding to all names starting with a dot and having more than two characters. The goal is to exclude .., i.e. the parent directory from the arguments.

Hypothetical files and directories like .a would still be missed, and those starting with a dash (-) would be at best ignored and at worst trigger an error.

If you have files or directories names starting with a dash or names starting with a dot followed by a single character, this enhanced version will properly report all directory sizes:

sudo du -csh -- .[^.] .??* *

Here the -- is telling du to take arguments starting with a dash as names, not options and .[^.] is adding to the processed names all file and directories starting with a dot followed by any single character but a dot. This encompass all possible names for files and directories.

Several shells provide custom workarounds to these naming issues:

bash:

shopt -s dotglob
sudo du -csh -- *

zsh:

sudo du -csh -- *(D)

ksh93:

FIGNORE='@(.|..)'
sudo du -csh -- *
9
  • ... and please don't try .* as you would go through .. (which will take a significantly larger amount of time).
    – John WH Smith
    Nov 5, 2014 at 12:07
  • 1
    @JohnWHSmith, not if you're using zsh (or fish) which have fixed that bug. In zsh, you'd use du -csh -- *(D) though to include dot file. You also need the -- as otherwise a file called -s for instance wouldn't be accounted for.
    – Stéphane Chazelas
    Nov 5, 2014 at 12:09
  • @StéphaneChazelas This shell always has something to surprise me... I'm guessing a more portable way to do this would be to use find and du: find -maxdepth 1 | du -csh.
    – John WH Smith
    Nov 5, 2014 at 12:10
  • @JohnWHSmith, more like find . ! -name . -prune -exec du -csh {} +. -maxdepth is not portable, find -print output is generally not post-processable and du doesn't read its stdin.
    – Stéphane Chazelas
    Nov 5, 2014 at 12:14
  • Thanks. As I say in the comment beneath my question, I'm seeing that du -ch (instead of du -csh) with no other args is returning the results I need. Isn't this a little over complicated or am I missing something?
    – TenLeftFingers
    Nov 5, 2014 at 12:14
2

Not necessarily the answer you're looking for, but something good to know about is:

sudo du -max /var/lib/jenkins | sort -n

That's probably one of the commands I use most often for this sort of thing. The "-a" includes total directory size (so you'll see a directory full of small files as one big directory in the final output, as well as seeing all the files), and the -x keeps it in in one filesystem. Using -m rather than -h makes it arguably harder to read, but using a fixed unit facilitates a more portable numeric sort (as in, still works when you don't have a pretty new sort command which supports the -h modifier).

0
1

Try out ncdu. This is an ncurses-based mc-like directory browser that will let you see which directories are most occupied and optionally delete the ones you're not interested in, as seen on this screenshot:

enter image description here

2
  • @TenLeftFingers: if you think that this answers your problem to a bigger degree than a currently accepted answer, please consider accepting this instead.
    – d33tah
    Nov 6, 2014 at 18:02
  • The accepted answer (unwieldy as the commands are) uses tools installed by default on all our servers and explains those so I will need to use that in practise. I have reduced it to a much simpler command, as per my comment but it did answer my question, technically speaking.
    – TenLeftFingers
    Nov 7, 2014 at 19:46
0

There are also nice graphical tools to help figure out what in is taking up lots of space in a directory.

I like Filelight which is part of KDE. Here it is exploring /var/cache on my computer. The green and the yellow are related to apt, the red is java, and the magenta is pbuilder.

filelight

The inner circles the immediate subdirectories and they are broken out into smaller pieces in the outer circles. This tool makes it very easy to browse around and see what is taking up lots of space -- not just large files, but large collections of small files as well.

1
  • I'm familiar with this great tool, but our servers are naturally headless.
    – TenLeftFingers
    Nov 7, 2014 at 19:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .