Httpd : no space left on device

Question

I am using Fedora 17 and over the last few days I am having an issue with my system. Whenever I try to start httpd it shows me:

Error: No space left on device

When I execute systemctl status httpd.service, I receive the following output:

httpd.service - The Apache HTTP Server (prefork MPM)
          Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
          Active: inactive (dead) since Tue, 19 Feb 2013 11:18:57 +0530; 2s ago
         Process: 4563 ExecStart=/usr/sbin/httpd $OPTIONS -k start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/httpd.service

I tried to Google this error and all links point to clearing the semaphores. I don't think this is the issue as I tried to clear the semaphores but that didn't work.

Edit 1

here is the output of df -g

[root@localhost ~]# df -h
Filesystem              Size  Used Avail Use% Mounted on
rootfs                   50G   16G   32G  34% /
devtmpfs                910M     0  910M   0% /dev
tmpfs                   920M  136K  920M   1% /dev/shm
tmpfs                   920M  1.2M  919M   1% /run
/dev/mapper/vg-lv_root   50G   16G   32G  34% /
tmpfs                   920M     0  920M   0% /sys/fs/cgroup
tmpfs                   920M     0  920M   0% /media
/dev/sda1               497M   59M  424M  13% /boot
/dev/mapper/vg-lv_home  412G  6.3G  385G   2% /home

Here is the deatail of httpd error log

[root@localhost ~]# tail -f /var/log/httpd/error_log
[Tue Feb 19 11:45:53 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Feb 19 11:45:53 2013] [notice] Digest: generating secret for digest authentication ...
[Tue Feb 19 11:45:53 2013] [notice] Digest: done
[Tue Feb 19 11:45:54 2013] [notice] Apache/2.2.23 (Unix) DAV/2 PHP/5.4.11 configured -- resuming normal operations
[Tue Feb 19 11:47:23 2013] [notice] caught SIGTERM, shutting down
[Tue Feb 19 11:48:00 2013] [notice] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Tue Feb 19 11:48:00 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Feb 19 11:48:00 2013] [notice] Digest: generating secret for digest authentication ...
[Tue Feb 19 11:48:00 2013] [notice] Digest: done
[Tue Feb 19 11:48:00 2013] [notice] Apache/2.2.23 (Unix) DAV/2 PHP/5.4.11 configured -- resuming normal operations
tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

Edit 2 here is the output of df-i

[root@localhost ~]# df -i
Filesystem               Inodes  IUsed    IFree IUse% Mounted on
rootfs                  3276800 337174  2939626   11% /
devtmpfs                 232864    406   232458    1% /dev
tmpfs                    235306      3   235303    1% /dev/shm
tmpfs                    235306    438   234868    1% /run
/dev/mapper/vg-lv_root  3276800 337174  2939626   11% /
tmpfs                    235306     12   235294    1% /sys/fs/cgroup
tmpfs                    235306      1   235305    1% /media
/dev/sda1                128016    339   127677    1% /boot
/dev/mapper/vg-lv_home 26984448    216 26984232    1% /home

Thanks

Answer 1

Here we see evidence of a problem:

tail: inotify resources exhausted

By default, Linux only allocates 8192 watches for inotify, which is ridiculously low. And when it runs out, the error is also No space left on device, which may be confusing if you aren't explicitly looking for this issue.

Raise this value with the appropriate sysctl:

fs.inotify.max_user_watches = 262144

(Add this to /etc/sysctl.conf and then run sysctl -p.)