Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Var is showing as full to many apps like Nagios, Puppet, and the LVM tools (pvs, vgs, etc)

df -h output

6.0G  4.3G  1.4G  77% /var

vgs output

/var/lock/lvm/V_rootvg:aux: open failed: No space left on device
  Can't get lock for rootvg
  Skipping volume group rootvg

lsof +L1 shows nothing under var anymore, so I don't think there are unlinked files which have yet to be cleared from the /var filesystem. I don't understand why 1.4G free on a 6G filesystem is considered full. I know some space is reserved by the system on each filesystem but that can't be it, it's too much space. The filesystem is ext3 on Red Hat 5.

dumpe2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          c8f44510-e8f7-4e2e-950a-1410b069910e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              393216
Block count:              1572864
Reserved block count:     78627
Free blocks:              1183083
Free inodes:              388144
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      63
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Mon Apr 29 13:12:02 2013
Last mount time:          Wed Oct 23 19:10:44 2013
Last write time:          Wed Oct 23 19:10:44 2013
Mount count:              6
Maximum mount count:      -1
Last checked:             Mon Apr 29 13:12:02 2013
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      8766dfd5-c802-4bc3-81cc-21869e810656
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             32M
Journal length:           8192
Journal sequence:         0x0112568e
Journal start:            3334
share|improve this question
4  
What is the output of df -i /var? –  Hauke Laging Apr 2 at 17:41
1  
@HaukeLaging fwiw the dumpe2fs shows 388,144 inodes free. –  Bratchley Apr 2 at 17:58
3  
@JoelDavis, I'm betting that count is wrong and df -i will show he is indeed out, and a fsck will fix the count in the superblock. –  psusi Apr 2 at 18:20
1  
@GreggLeventhal if you extend the filesystem you'll get more inodes available. Since you're on LVM you can do it without unmounting the filesystem with lvextend -r Alternatively, you can see if there are files you can unlink. –  Bratchley Apr 2 at 19:49
1  
@GreggLeventhal Delete a few old files in /var/log or move them to tmpfs or another volume. –  Hauke Laging Apr 2 at 20:37

2 Answers 2

up vote 1 down vote accepted

Looking at the comments others have helped you diagnose you're out of inodes. If you need to make a few available so you can get some basic access back to your system then you could delete the following files on a CentOS 5 install, assuming you can live without them.

Example

$ sudo rm -fr /var/log/*.[1-9]?(.gz)

This will remove any of the previously backed up files in /var/log. This should buy you a few dozen inodes to start.

Counting inodes

using df

I usually use the command df to determine the number available.

$ df -i /
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     59932672  807492 59125180    2% /
using tune2fs

You can also use tune2fs. With it you'll need to give it the path to the LVM LV mapper.

$ tune2fs -l /dev/mapper/VolGroup00-LogVol00 | grep -i inode
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Inode count:              59932672
Free inodes:              59126861
Inodes per group:         32768
Inode blocks per group:   1024
First inode:              11
Inode size:       128
Journal inode:            8
First orphan inode:       21561629
Journal backup:           inode blocks

Freed up some inodes, now what?

With some breathing room you basically have a couple of options.

  1. I would start by trying to quickly get a list together of files that can be targeted for deletion, so you can begin to get more headroom. I'd focus on /tmp and /var some more potential files to be removed.

  2. If you have old versions of Java or anything installed under /usr/local or /opt I'd pick on those next.

  3. I'd start formulating a list of installed RPMs that can be uninstalled

  4. If you've been using YUM to do updates on this server you can clear out its cache.

    $ sudo yum clean all
    
  5. Look into adding additional space.

share|improve this answer
    
How about /var/log/*.[1-9]?(.gz) for matching old log files (extglob required). –  Graeme Apr 2 at 23:43
    
@Graeme - yes that's better, I'll add it, thanks. Careful or ppl will know that we've run out of inodes before 8-) –  slm Apr 2 at 23:47

One likely reason for running out of inodes is that a large number of files has accumulated in a particular directory for whatever reason. You could check the usual suspects, eg /tmp, /var/tmp, /var/log etc. If you don't find anything, here is a command that I have cobbled together to list the top 50 directories in the filesystem containing the most files/directories in their first level.

find / -xdev -type d -exec sh -c '
  num=$(find "$0" -maxdepth 1 | wc -l); echo "$num $0"' {} \; |
  sort -n |
  tail -50

Note the top level of mount points are included as well - this is not trivial to exclude.

share|improve this answer
    
It was the pnp4nagios directory under /var/spool/ it creates a very large number of small files. I ended up creating a logical volume and mounting it directly to this directory. This resolved my issues. –  Gregg Leventhal Apr 3 at 19:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.