5

On my Ubuntu 12.04 servers, I need to write an init.d script that waits until the infiniband interface (device: mlx4_0, interface: ib0) is completely up, and until the knem kernel module is loaded.

I have an init.d script that works if I wait until the system is up and manually service myscript start, but it does not work if I load it normally at boot time. I'm using boot order 99, but it does not start correctly because I need it to wait until those features are running.

What is the correct syntax in an init.d to achieve that (one kernel module and )? I guess there must be some keyword like # Required-Start: $remote_fs $syslog $network but I can't find the correct ones for specific interface and kernel modules.

For background: the init script is related to SLURM, openMPI and infiniband interactions. I have compiled SLURM with Mellanox infiniband driver support, and openMPI is linked to this version of SLURM. The result is that openMPI is using directly the mellanox infiniband driver, which is much more powerful than ipob (ip over inifiniband). To do that, it needs to use the registered memory of the system, which has to be set as unlimited.

SO:

I have added a few logger outputs in my init.d script. I notice that in fact the modules are up and running. So I don't understand fully the problem. It is strange, maybe related to some environment variables needed and that are set up only in a full user space and not at init time.

The problem concerns the memlock limits set in /etc/security/limits.conf. To make it work I had to set

*             -   memlock       unlimited
root          -   memlock       unlimited

and in this way, when I launch the slurm deamon with an ssh connection everything works, and when it is the init process that starts the daemon, it is like it does not consider the rules in /etc/security/limits.conf.

  • Danduk82 - it's probably a good idea to put the answer portion of your question in an answer if you can. You should feel free to accept your own answer as well. If you do this thing and ping me, I'll likely upvote it. – mikeserv Aug 6 '14 at 3:50
  • 1
    @mikeserv : actually, this is not the answer, it is part of the question. The rules about memlock seem to work only if I start the deamon with a ssh connection, and it does not work when it is started by init. – Danduk82 Aug 7 '14 at 12:08
  • ahhh... That makes sense. It went over my head before. – mikeserv Aug 7 '14 at 14:41
3

I don't know specifically about an init.d file, but a udev rule for running a script upon device-add might look like:

ACTION=="add", ATTRS{idVendor}=="VID", ATTRS{idProduct}=="PID", RUN+="/path/to/executable"

You should do some digging in udevadm to learn more about the way the device is typically added and its modules loaded. You'll also find the correct values for VID and PID in there.

Does it accept a combination of both device and kernel module?

Yes. Well... no. Maybe? The answer to this question entirely depends on the level at which you intercept the device-add. udev has to do many things from the point at which it initially detects a device to the point at which it has fully loaded and initialized the device and it can be considered up.

Some of these things are likely:

  • initially detecting the hardware on its parent bus/device/subsystem

  • finding and loading the appropriate kernel module

  • populating the /dev devfs file system with the appropriate device special file

  • detecting any child-devices the current device adds and rinse, repeat

You can specify a rule for all or any of these levels of action. You can also udevadm trigger or monitor or several other things directly in real-time to determine exactly what these levels of action might be.

I recommend you look closely at this information if you wish to become more familiar with it.

  • 1
    Interesting, does it accept a combination of both device and kernel module? Note that the kernel module (knem) is not related to the infiniband device but to locked memory. So they are 2 distinct pieces of software, the infiniband drivers are loaded when the device comes up. I'll see if I can find something that works with udev. Thanks for the answer. – Danduk82 Aug 5 '14 at 9:08
1

Use an Upstart job for this:

# Ensures that the device is up and filesystem is too
start on filesystem and net-device-up IFACE=ib0
stop on runlevel [016]

# Ensures that module is loaded
pre-start exec modprobe -q knem

exec /path/to/exec
  • I have tried your suggestion. It does not change. Actually the module is already correctly loaded. The problem seems to be related with permissions or (more plausibely) with environment variables. Please refer to the edit in my question. – Danduk82 Aug 7 '14 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.