Hang during 'touch' operation on NFS mount

Question

I have two NFS clients mounted to an openfiler 2.99 NFS share at 192.0.2.3:

192.0.2.1 mounts 192.0.2.3:/mnt/nfs01/volnfs01/share01 with rw,noatime,nodiratime,hard,rsize=32768,wsize=32768,noacl,nocto,tcp,nfsvers=3
192.0.2.1 mounts 192.0.2.3:/mnt/nfs01/volnfs01/share02 with rw,noatime,nodiratime,hard,rsize=32768,wsize=32768,nfsvers=3,tcp,noacl,nocto
192.0.2.2 mounts 192.0.2.3:/mnt/nfs01/volnfs01/share02 with rw,noatime,nodiratime,hard,rsize=32768,wsize=32768,nfsvers=3,tcp,noacl,nocto

`touch` broken

My problem is with 192.0.2.2's NFS mount. When I touch a file in on that mount, the process hangs indefinitely... I used strace touch /mnt/share02/this and got this far...

rt_sigaction(SIGRTMIN, {0x3b71c05ae0, [], SA_RESTORER|SA_SIGINFO, 0x3b71c0f500}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x3b71c05b70, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x3b71c0f500}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM_INFINITY}) = 0
brk(0)                                  = 0xafb000
brk(0xb1c000)                           = 0xb1c000
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fce244c0000
close(3)                                = 0
open("/mnt/share02/this", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666
                                                                    ^^^ stops touching
                                                                     |
                                                                     |

When I check ps -elf from another terminal, I see the process in "D" state...

[mpenning@host192_0_2_2 ~]$ ps -elf | awk '$2=="D"'
0 D mpenning  8157  8032  0  80   0 - 26293 rpc_wa 09:59 pts/2    00:00:00 touch /mnt/share02/this
[mpenning@host192_0_2_2 ~]$

showmount isn't finding a problem though....

[mpenning@host192_0_2_2 ~]$ showmount -e 192.0.2.3
Export list for 192.0.2.3:
/mnt/nfs01/volnfs01/share01 192.0.2.2/255.255.255.255,192.0.2.1/255.255.255.255
/mnt/nfs01/volnfs01/share02 192.0.2.2/255.255.255.255,192.0.2.1/255.255.255.255
[mpenning@host192_0_2_2 ~]$

Various status of NFS services...

[mpenning@host192_0_2_2 ~]$ service nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 9168) is running...
nfsd (pid 9232 9231 9230 9229 9228 9227 9226 9225) is running...
rpc.rquotad (pid 9164) is running...
[mpenning@host192_0_2_2 ~]$ service rpcbind status
rpcbind (pid  9088) is running...
[mpenning@host192_0_2_2 ~]$ service nfslock status
rpc.statd (pid  9256) is running...
[mpenning@host192_0_2_2 ~]$

Network configuration (default gw isn't required since this is a dedicated layer2 NFS vlan):

[mpenning@host192_0_2_2 ~]$ sudo cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
NM_CONTROLLED=no
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.0.2.2
NETMASK=255.255.255.0
DNS2=none
TYPE=Ethernet
GATEWAY=
DNS1=none
IPV6INIT=no
USERCTL=no
MTU=9000
[mpenning@host192_0_2_2 ~]$

This looks pretty nasty. I have done the following on 192.0.2.2:

Restarted all NFS
init 6 the machine
ping 192.0.2.3 to make sure it still has connectivity to the server
Checked dmesg
Checked showmount -e 192.0.2.3

This feels like a permissions problem, but I don't know where to go from here...

How can I fix this problem so I can read / write to any file on 192.0.2.2's mount of 192.0.2.3:/mnt/nfs01/volnfs01/share02?

`touch` works

If I execute the same touch command from 192.0.2.1, everything is fine...

rt_sigaction(SIGRTMIN, {0xb096e0, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0xb09b80, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="host192_0_2_1.localdomain.local", ...}) = 0
brk(0)                                  = 0x8d4d000
brk(0x8d6e000)                          = 0x8d6e000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=99158544, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7574000
close(3)                                = 0
open("/mnt/share02/this", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK|O_LARGEFILE, 0666) = 3
dup2(3, 0)                              = 0
close(3)                                = 0
utimensat(0, NULL, NULL, 0)             = 0
close(0)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?

`/etc/exports` from 192.0.2.3

[root@T1-Netfile01 backups]# head /etc/exports

# PLEASE DO NOT MODIFY THIS CONFIGURATION FILE!
#       This configuration file was autogenerated
#       by Openfiler. Any manual changes will be overwritten
#       Generated at: Fri Nov 8 9:35:39 CST 2013

/mnt/nfs01/volnfs01/share02 192.0.2.1/255.255.255.255(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.0.2.2/255.255.255.255(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)

/mnt/nfs01/volnfs01/share01 192.0.2.1/255.255.255.255(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.0.2.2/255.255.255.255(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)

[root@T1-Netfile01 backups]#

@slm, Interesting... the first time I try the touch, I get touch: cannot touch /mnt/share02/this': Permission denied. When I try to ls /mnt/share02`, the command hangs. — Mike Pennington, Nov 8, 2013 at 16:15
Are we sure that uid 96 is consistent across systems? Also take the 255... business out, I don't think you need those. — slm, Nov 8, 2013 at 16:17
My showmounts are typically like this: /cobbler/isos 192.168.1.0/24. I'd change those 255...'s to a /32. They might be your issue, but hopeful but doubtful. — slm, Nov 8, 2013 at 16:25
Also are you sure that the nfslock and related services are running on the .2.2 machine? — slm, Nov 8, 2013 at 16:27
uid 96 was indeed missing on host192_0_2_2; however, that didn't change things; I still get a hang when I ls /mnt/share02 — Mike Pennington, Nov 8, 2013 at 16:34

Mike Pennington · Accepted Answer · 2013-11-08 18:43:01Z

3

And if you change the order of the IPs in the /etc/exports file what happens then? Put the .2.2 IP 1st and the .2.1 2nd.

Also I would confirm what the exports are presenting as using the command:

$ showmount -e 192.0.2.3

/etc/exports can be very particular about the formatting!

Other things to try

I typically specify my hosts in the /etc/exports like this:

/cobbler/isos   192.168.1.0/24(rw,no_root_squash)

So for you with a single host IP:

/mnt/nfs01/volnfs01/share02 192.0.2.1/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.0.2.2/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)
/mnt/nfs01/volnfs01/share01 192.0.2.1/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)  192.0.2.2/32(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)

nfs related servcies

Make sure that nfslock and other related services are both running on 192.0.2.2.
If you're using jumbo frames, be sure that ping -s <jumbo_mtu> 192.0.2.3 works from 192.0.2.2

edited Nov 8, 2013 at 18:43

Mike Pennington

2,4424 gold badges30 silver badges40 bronze badges

answered Nov 8, 2013 at 16:15

slm♦

358k114 gold badges756 silver badges863 bronze badges

showmount doesn't highlight a problem... I updated my question
– Mike Pennington
Nov 8, 2013 at 16:17
btw, I also changed the netmasks on that mount, and also checked the related NFS services... see the latest update to my question.
– Mike Pennington
Nov 8, 2013 at 16:46
The problem was jumbo frames. I found a link in the layer2 network that was still using 1500B MTUs, but I gave you credit for the solution since I also had permission problems on 192.0.2.2; perhaps you should add a bit about uid / gid 96 in your answer
– Mike Pennington
Nov 8, 2013 at 18:42
You can include only two addresses with a netmask of 30 /mnt/nfs01/volnfs01/share01 192.0.2.1/30(rw,anonuid=96,anongid=96,secure,root_squash,wdelay,sync)
– h4unt3r
Sep 26, 2014 at 23:47

Add a comment |

Stack Exchange Network

Hang during 'touch' operation on NFS mount

`touch` broken

`touch` works

`/etc/exports` from 192.0.2.3

1 Answer 1

Other things to try

Your Answer

Not the answer you're looking for? Browse other questions tagged
filesystems
files
nfs
or ask your own question.

Hot Network Questions

Hang during 'touch' operation on NFS mount

touch broken

touch works

/etc/exports from 192.0.2.3

1 Answer 1

Other things to try

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged filesystemsfilesnfs or ask your own question.

Related

Hot Network Questions

`touch` broken

`touch` works

`/etc/exports` from 192.0.2.3

Not the answer you're looking for? Browse other questions tagged
filesystems
files
nfs
or ask your own question.