Var is showing as full to many apps like Nagios, Puppet, and the LVM tools (pvs, vgs, etc)
df -h output
6.0G 4.3G 1.4G 77% /var
vgs output
/var/lock/lvm/V_rootvg:aux: open failed: No space left on device
Can't get lock for rootvg
Skipping volume group rootvg
lsof +L1
shows nothing under var anymore, so I don't think there are unlinked files which have yet to be cleared from the /var
filesystem. I don't understand why 1.4G free on a 6G filesystem is considered full. I know some space is reserved by the system on each filesystem but that can't be it, it's too much space. The filesystem is ext3
on Red Hat 5.
dumpe2fs 1.41.12 (17-May-2010)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: c8f44510-e8f7-4e2e-950a-1410b069910e
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 393216
Block count: 1572864
Reserved block count: 78627
Free blocks: 1183083
Free inodes: 388144
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 63
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Filesystem created: Mon Apr 29 13:12:02 2013
Last mount time: Wed Oct 23 19:10:44 2013
Last write time: Wed Oct 23 19:10:44 2013
Mount count: 6
Maximum mount count: -1
Last checked: Mon Apr 29 13:12:02 2013
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 8766dfd5-c802-4bc3-81cc-21869e810656
Journal backup: inode blocks
Journal features: journal_incompat_revoke
Journal size: 32M
Journal length: 8192
Journal sequence: 0x0112568e
Journal start: 3334
df -i /var
?dumpe2fs
shows 388,144 inodes free.lvextend -r
Alternatively, you can see if there are files you can unlink./var/log
or move them to tmpfs or another volume.