7

I am seeing processes on Linux with the name "[123.45.78.901-ma]", where the number is the IP address of our NFS server. Does anyone know what these processes are, where I can find out information about them, or a good keyword to search the web with?

Background:

I see the processes on a machine where something funky is going on with NFS - something seems to be stuck in a loop making a huge number of NFS requests (read, renew, and authrefrsh), but all obvious candidates (like user programs) that might be doing so have been killed.

A large number of [123.45.78.901-ma] processes exist (for brief periods of time) on the machine in question, but not (at least not for long enough that I have detected them) on machines that are acting normally.

So it would appear that these processes are involved somehow in the problem, but I don't know how.

I found them while looking for processes in "uninterruptible sleep", via

while [ 1 ] ; do ps -w -w -e -o pid,state,tname,time,args| grep ' D ' | grep -v grep; sleep 0.1; done

Details about the system:

  • Red Hat Enterprise Linux Workstation release 6.3 (Santiago)

  • Linux [...] 2.6.32-279.5.1.el6.x86_64 #1 SMP Tue Jul 24 13:57:35 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Thanks in advance!

2 Answers 2

4

Those are kernel threads which are likely to be NFS mount helpers, which only appear when a new NFS mount is being mounted on a client.

You can see what they are sleeping on if you can look in /proc/<pid>/stack when they are in the uninterruptible sleep state, which gives a backtrace of what they are doing. This will require root privileges though, which by the sounds of your comments on the other answer isn't something you have in this case.

If you are seeing these kernel threads on this host but not others which are set up in a similar way then this implies that the RPC call performance on this host is lower than the other hosts for some reason. Perhaps there are other network problems which are causing this to show up?

For info, I had a similar finding when investigating NFS mounts hanging on otherwise-working hosts. In my case it appeared to be due to a kernel bug in the particular kernel version the hosts were running at the time, causing the [123.456.78.90-ma] kernel threads to become permanently stuck in the uninterruptible sleep state.

6
  • I have similar problems right now. Only the letter "a" is missing, so the pattern is [123.456.78.90-m]. Do you remember the kernel version which caused the trouble for you? How did you find out that it is a kernel bug? Thanks!
    – azimut
    Commented Feb 23, 2016 at 9:45
  • It being a kernel bug was a guess -- it looks very much like the NFS processes get stuck in kernel-space (I don't have it to hand now but there was a kernel trace in the logs if I remember correctly...). Also I had several machines with slightly different kernel versions and only one particular version showed up this problem for me. I don't have the exact kernel version noted either, but it was happening for me on CentOS 7 hosts with a couple of patch-levels behind the latest kernel version at the time... Sorry nothing more specific than that! :s Commented Feb 23, 2016 at 12:01
  • (If I'd managed to make it repeatable and it wasn't on something in production then I would have investigated more and put a bug in, but I didn't have specific enough info to do so at the time) Commented Feb 23, 2016 at 12:02
  • Thanks for the details. My clients are on Ubuntu linux-image-4.2.0-27-generic, and sporadically, suddenly some NFS mount gets stuck with those [123.456.78.90-m] threads in uninterruptible disk sleep. Might happen after 10 minutes or only after a few days. Often, then also some mount.nfs process is in disk sleep, which was initiated by the automounter. Everything used to work fine till maybe ~1 month ago. I'm quite cueless how to further debug this. My hope now is that the recent update to linux-image-4.2.0-30-generic will solve it.
    – azimut
    Commented Feb 23, 2016 at 12:38
  • CentOS 7 with kernel 3.10.0-514.10.2.el7.x86_64 has done this to me on one host again. Hasn't happened for ages though, so it's rather difficult to pin down to a particular kernel or other software version, or any particular level of load. I do use autofs to automount NFS shares for user home directories though, which may put more pressure on the mount/unmount parts of NFS... Commented Nov 17, 2017 at 16:05
3

On Linux, ps reports kernel threads in square brackets. These are not "processes" in the normal sense of that word. That is, there is not an executable loaded from disk to start them, they aren't owned by a normal user, etc. They're just one of the many things the kernel has going on in the background.

For that reason, the name shown by ps doesn't have to correspond to any file on your hard disk. (In the case of zombie processes, though, it does.)

5
  • Warren: Thanks, yes. Do you know how I could find specific details about these ones though? The name isn't easy to Google...
    – N F
    Commented Feb 3, 2013 at 20:29
  • 1
    Because they're kernel threads and not processes, there's not exactly a man page to peruse to figure this out. You'd have to know how the Linux NFS implementation works internally. Personally, I wouldn't be chasing a problem like that through the process table. I'd be using Wireshark instead. Commented Feb 3, 2013 at 20:32
  • 1
    Indeed, unfortunately I am just a normal user on this system so I can't really use Wireshark and I'm trying to make as much progress as I can this way.
    – N F
    Commented Feb 3, 2013 at 20:45
  • If you're not the system's administrator, you're not in any position to do anything about NFS problems anyway. Commented Feb 3, 2013 at 21:01
  • Well, I can write them obnoxious emails :)
    – N F
    Commented Feb 3, 2013 at 21:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .