File descriptor limits are lost after a system reboot

Question

I'm trying to increase the default file descriptor limits for processes on my system. Specifically I'm trying to get the limits to apply to the Condor daemon and its sub-processes when the machine boots. But the limits are never applied on machine boot.

I have the limits set in /etc/sysctl.conf:

[root@mybox ~]# cat /etc/sysctl.conf
# TUNED PARAMETERS FOR CONDOR PERFORMANCE
# See http://www.cs.wisc.edu/condor/condorg/linux_scalability.html for more information

# Allow for more PIDs (to reduce rollover problems); may break some programs
kernel.pid_max = 4194303

# increase system file descriptor limit
fs.file-max = 262144

# increase system IP port limits
net.ipv4.ip_local_port_range = 1024 65535

And in /etc/security/limits.conf:

[root@mybox ~]# cat /etc/security/limits.conf
# TUNED PARAMETERS FOR CONDOR PERFORMANCE
# See http://www.cs.wisc.edu/condor/condorg/linux_scalability.html for more information
# Increase the limit for a user continuously by editing etc/security/limits.conf.
*        soft  nofile        32768
*        hard  nofile      262144 #65536

The trouble I run in to is, on system reboot, the limits don't seem to apply to Condor and its processes. After a reboot, if I look at the file descriptor limit for a Condor process I see:

[root@mybox proc]# cat /proc/`/sbin/pidof condor_schedd`/limits | grep 'Max open files'
Max open files            1024                1024

But if I restart the condor_schedd process after a reboot the limits are increased as expected:

[root@mybox proc]# cat /proc/`/sbin/pidof condor_schedd`/limits | grep 'Max open files'
Max open files            32768                262144

The boot.log indicates these limits are being set before my Condor daemon and its processes are being started:

May 18 07:51:52 mybox sysctl: net.ipv4.ip_forward = 0
May 18 07:51:52 mybox sysctl: net.ipv4.conf.default.rp_filter = 1
May 18 07:51:52 mybox sysctl: net.ipv4.conf.default.accept_source_route = 0
May 18 07:51:52 mybox sysctl: kernel.sysrq = 0
May 18 07:51:52 mybox sysctl: kernel.core_uses_pid = 1
May 18 07:51:52 mybox sysctl: kernel.pid_max = 4194303
May 18 07:51:52 mybox sysctl: fs.file-max = 262144
May 18 07:51:52 mybox sysctl: net.ipv4.ip_local_port_range = 1024 65535
May 18 07:51:52 mybox network: Setting network parameters: succeeded
May 18 07:51:52 mybox network: Bringing up loopback interface: succeeded
May 18 07:51:57 mybox ifup: Enslaving eth0 to bond0
May 18 07:51:57 mybox ifup: Enslaving eth1 to bond0
May 18 07:51:57 mybox network: Bringing up interface bond0: succeeded
May 18 07:52:17 mybox hpsmhd: smhstart startup succeeded
May 18 07:52:17 mybox condor: Starting up Condor
May 18 07:52:17 mybox rc: Starting condor:  succeeded
May 18 07:52:17 mybox crond: crond startup succeeded

Obviously I'd like to avoid having to boot a machine and then restart process that I need these increased limits to apply to -- what have I done wrong that's preventing these limits from applying to the processes when the machine boots?

Answer 1

Add ulimit -n 262144 to the condor init script.