5

I interconnect several of my Linux servers with NFS. For a while its been going fine but for some reason keeps getting more and more unstable

Example: Copy a 4 GB file from client to NFS server, fails before finishing with an IO error. dmesg is filled with

[1525450.884071] nfs: server quackron not responding, timed out
[1525451.384072] nfs: server quackron not responding, timed out
[1525451.884072] nfs: server quackron not responding, timed out
[1525452.384086] nfs: server quackron not responding, timed out
[1525452.884065] nfs: server quackron not responding, timed out
[1525584.112127] nfs: server quackron not responding, timed out
[1525669.492066] nfs: server quackron not responding, timed out
[1525930.544062] nfs: server quackron not responding, timed out

Mount options (the many extra options are to prevent a disappearing NFS server from locking the whole directory until a restart or umount trickery)

rw,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5,vers=4,addr=10.0.0.20,clientaddr=10.0.0.1

Going to the server, I get this in dmesg

[1384141.237197] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[1384141.237237] NFSD: starting 90-second grace period
[1439491.137710] nfsd: last server has exited, flushing export cache
[1439492.461197] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[1439492.461236] NFSD: starting 90-second grace period

I can't find it in this dmesg or the log, but I do know I've seen "nfsd last server has exited signal 15" errors. Googling can't seem to find any useful information

Any idea's what would cause nfs to constantly crash?

Note: This server is running Ubuntu Server 11.10 and is fully up to date

  • Have you tried scp to copy the same file? – Karlson Feb 23 '12 at 21:29
  • @Karlson Just tried it. Works perfectly fine. Right afterwards tried NFS just crashed again – TheLQ Feb 24 '12 at 21:16
  • @TheLQ - Please input the defined NFS export from /etc/exports. – 2bc Mar 23 '12 at 0:28
  • Its not in /etc/exports, its mounted in a script for various reasons. Script command: sudo mount -t nfs -o intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 -v -v -v $1:$2 $3 with the variables at the end corresponding to server, remote folder, and local folder – TheLQ Mar 23 '12 at 1:33
  • is this on 32 or on 64 bits ? – Coren Mar 23 '12 at 7:56
3
+50

As a test, modify your mount on the client side to:

sudo mount -t nfs -o intr,hard,timeo=14,rsize=8192,wsize=8192 

or, if you put it in fstab on the client side:

remoteserver:/remote/path /local/path nfs rsize=8192,wsize=8192,hard,timeo=14,intr 0 0 

I use these settings at work and we routinely transfer several TB a week.

Use these settings adding none of the others except for your specific pathing variables.

  • Just did some testing, seems to of fixed the problem! Connection is stable enough to copy a 3.6 GB ISO without disconnecting. Seems to even be a bit faster, although I'm not for sure. Thanks! – TheLQ Mar 26 '12 at 21:13
  • Awesome!! You can play with the rsize and wsize this modifies the data transfer buffer sizes. The only thing you need to worry about is your network setup as it relates to MTU I have never had an issue but, others have. Without going and looking I think you can go up to 32,xxx for them. – 2bc Mar 26 '12 at 21:22
  • 1
    Does anyone have an idea what is actually causing this issue? The default NFS mounting set hard, negotiate rsize and wsize to max value. Is it because of intr? Or it is because of a much smaller timeo time (1.4s compared to default 60s). – Isaac Nov 15 '14 at 11:45
0

If it's a dedicated NFS VLAN, double check you don't have any duplicate IPs on your network. I just debugged an issue very much like this, and it turns out there were some typos in kickstart files and out of 40 hosts, we had 5 that were using duplicate IP addresses. NFS would work great and copies would be fast for a while, but performance would gradually taper off, then hang altogether.

  • Went ahead and verified that there was no duplicate IP's, couldn't find any – TheLQ Mar 26 '12 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.