Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am seeing processes on Linux with the name "[123.45.78.901-ma]", where the number is the IP address of our NFS server. Does anyone know what these processes are, where I can find out information about them, or a good keyword to search the web with?

Background:

I see the processes on a machine where something funky is going on with NFS - something seems to be stuck in a loop making a huge number of NFS requests (read, renew, and authrefrsh), but all obvious candidates (like user programs) that might be doing so have been killed.

A large number of [123.45.78.901-ma] processes exist (for brief periods of time) on the machine in question, but not (at least not for long enough that I have detected them) on machines that are acting normally.

So it would appear that these processes are involved somehow in the problem, but I don't know how.

I found them while looking for processes in "uninterruptible sleep", via

while [ 1 ] ; do ps -w -w -e -o pid,state,tname,time,args| grep ' D ' | grep -v grep; sleep 0.1; done

Details about the system:

  • Red Hat Enterprise Linux Workstation release 6.3 (Santiago)

  • Linux [...] 2.6.32-279.5.1.el6.x86_64 #1 SMP Tue Jul 24 13:57:35 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

Thanks in advance!

share|improve this question

On Linux, ps reports kernel threads in square brackets. These are not "processes" in the normal sense of that word. That is, there is not an executable loaded from disk to start them, they aren't owned by a normal user, etc. They're just one of the many things the kernel has going on in the background.

For that reason, the name shown by ps doesn't have to correspond to any file on your hard disk. (In the case of zombie processes, though, it does.)

share|improve this answer
    
Warren: Thanks, yes. Do you know how I could find specific details about these ones though? The name isn't easy to Google... – N F Feb 3 '13 at 20:29
1  
Because they're kernel threads and not processes, there's not exactly a man page to peruse to figure this out. You'd have to know how the Linux NFS implementation works internally. Personally, I wouldn't be chasing a problem like that through the process table. I'd be using Wireshark instead. – Warren Young Feb 3 '13 at 20:32
1  
Indeed, unfortunately I am just a normal user on this system so I can't really use Wireshark and I'm trying to make as much progress as I can this way. – N F Feb 3 '13 at 20:45
    
If you're not the system's administrator, you're not in any position to do anything about NFS problems anyway. – Warren Young Feb 3 '13 at 21:01
    
Well, I can write them obnoxious emails :) – N F Feb 3 '13 at 21:09

Those are kernel threads which are likely to be NFS mount helpers, which only appear when a new NFS mount is being mounted on a client.

You can see what they are sleeping on if you can look in /proc/<pid>/stack when they are in the uninterruptible sleep state, which gives a backtrace of what they are doing. This will require root privileges though, which by the sounds of your comments on the other answer isn't something you have in this case.

If you are seeing these kernel threads on this host but not others which are set up in a similar way then this implies that the RPC call performance on this host is lower than the other hosts for some reason. Perhaps there are other network problems which are causing this to show up?

For info, I had a similar finding when investigating NFS mounts hanging on otherwise-working hosts. In my case it appeared to be due to a kernel bug in the particular kernel version the hosts were running at the time, causing the [123.456.78.90-ma] kernel threads to become permanently stuck in the uninterruptible sleep state.

share|improve this answer
    
I have similar problems right now. Only the letter "a" is missing, so the pattern is [123.456.78.90-m]. Do you remember the kernel version which caused the trouble for you? How did you find out that it is a kernel bug? Thanks! – azimut Feb 23 at 9:45
    
It being a kernel bug was a guess -- it looks very much like the NFS processes get stuck in kernel-space (I don't have it to hand now but there was a kernel trace in the logs if I remember correctly...). Also I had several machines with slightly different kernel versions and only one particular version showed up this problem for me. I don't have the exact kernel version noted either, but it was happening for me on CentOS 7 hosts with a couple of patch-levels behind the latest kernel version at the time... Sorry nothing more specific than that! :s – David Gardner Feb 23 at 12:01
    
(If I'd managed to make it repeatable and it wasn't on something in production then I would have investigated more and put a bug in, but I didn't have specific enough info to do so at the time) – David Gardner Feb 23 at 12:02
    
Thanks for the details. My clients are on Ubuntu linux-image-4.2.0-27-generic, and sporadically, suddenly some NFS mount gets stuck with those [123.456.78.90-m] threads in uninterruptible disk sleep. Might happen after 10 minutes or only after a few days. Often, then also some mount.nfs process is in disk sleep, which was initiated by the automounter. Everything used to work fine till maybe ~1 month ago. I'm quite cueless how to further debug this. My hope now is that the recent update to linux-image-4.2.0-30-generic will solve it. – azimut Feb 23 at 12:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.