targeted network speed over LAN

Question

I have a Solaris 11 system which has several NFS exports which are accessed my other systems within my LAN. I use a Linux system as a client for testing.

I wrote a quick script to test read speed and I average at around 110Mbps (or 13MB/s) on a Gigabit LAN. And I would have think it could get much faster. SSH (scp) only gives me 3.8MB/s but that's with encryption.

http gives me 11.5M/s, similar to NFS than. ain't this low?

what could be the bottleneck from these numbers?

Answer 1

NFS can't really maximise the throughput, because the client keeps sending please send me this much data to the server (this much being limited to a few kilobytes) and waits for the full answer before asking for more which means dead times, when all queues are empty. All FS over the network (CIFS, SSHFS) have the same kind of issue (and IIRC scp as well, or maybe it's only sftp, I can't remember)

Beside the encryption overhead, ssh also has some more performance limitations (see here for details).

HTTP, unless you use a client that performs chunked requests, should be straight TCP, so shouldn't have this kind of limitation. TCP should use its congestion control algorithm to maximize the throughput while avoiding congestion. While the first few kilobytes may be transferred slowly, you should be able to maximize your bandwidth within a few 10th of seconds if the two machines are connected via the same switch. It's possible that there be some poor network quality (like the odd packet loss).

Things you may want to try:

• Transfer the content of /dev/zero over a plain TCP connection (using socat or nc for instance), to rule out the bottle neck being FS access.
• Check your network interface statistics for transmission errors, and TCP stack statistics (netstat -s)
• Test with iperf (both TCP and UDP).

Answer 2

I had a similar situation in the past, and my way to solve was to force Solaris and Linux to mount NFS v3 instead of v4.

From the Solaris side you can edit the /etc/default/nfs and set the variables for the server side to server and accept only NFS v3 side.

Can't remember the variables names but it is self explanatory.

The test you've done is copying a single big file or several files? If it is several files I wouldn't be surprise but if it is a single file than yes.

Also, what kind of underlying storage do you have? single disk? raid setup? this is also impact your performance.

Answer 3

How fast can the underlying filesystem supply data? NFS itself can move data more quickly than you're seeing, which suggests there are other bottlenecks. You can convince yourself of this by using iperf to confirm the network can stream higher bandwidth.

I would log on to the solaris system and copy a file from the filesystem you are exporting to /dev/null to see how fast you can stream it. Note that you'll need to make some effort to not do this read test of a file that is already in the filesystem cache. With other filesystems, you can umount/mount to invalidate the cache, but that alone can be insufficient for ZFS.

Answer 4

You have to be aware of the capabilities of each piece of your stack. What kind of disks make up the filesystem for the NFS share? SAS? SATA? SSD? What type and size, and rpm speed of each? rotational latency? Is it a 4k drive? ZFS is optimized for 512-byte drives, and there are known issues with performance using new "advanced format" disks.

What size is your I/O workload? That also affects how fast your storage will appear.

From a purely storage perspective, these questions will affect how fast data can be written to and read from the backend disks. For example, let's say your NFS share was coming from a filesystem on a 7200 RPM 1TB SATA disk, and that your workload size is 64KB and is mostly random.

Random Workload, 64 KB IO size
--------------------------------------------------------------------------------
Average Read Seek Time:        8.4 ms
Average Write Seek Time:       8.9 ms
Rotational latency:           .120 ms
Average Latency:             4.166 ms
Average Read Service Time:  12.566 ms
Average Write Service Time: 13.066 ms
--------------------------------------------------------------------------------
Best Case Read IOPs:           79.000
Best Case Write IOPs:          76.000
--------------------------------------------------------------------------------
Best Case Read Throughput:  4.937 MB/s
Best Case Write Throughput: 4.750 MB/s
--------------------------------------------------------------------------------
Real World Read IOPs:         55.300
Real World Write IOPs:        53.200
--------------------------------------------------------------------------------
Real World Read Throughput:  3.325 MB/s
Real World Write Throughput: 3.325 MB/s
--------------------------------------------------------------------------------

That will give you an idea of the real world performance you can expect to get from a single disk at a certain workload size.

You can boost that performance by having a write-cache, or mounting your NFS filesystem asynchronously, so that write confirmations are not required before the next I/O begins (not that I recommend that, as it puts your data at risk).

Once you understand the performance of your backend storage, then you can start to look at additional overhead created by NFS (is it v3? v4? tcp? udp?), or SSH (which encryption algorithm are you using? you can use a lighter algorithm like arcfour to improve performance), or the network itself. Do you have a cheap-o 1GbE NIC? or a higher-end server quality NIC? are you sharing that nic between multiple services? is anything else generating traffic on your network? How about network congestion, collisions, or high retransmission rates?

All of these add up together to reduce even further the apparent performance of your backend storage, which really means that they all add up to make things appear very slow on the client side.

Now, the numbers above are a kind of worst case scenario, right? 100% random synchronous workload. If you have a sequential workload, or you can raise your IO workload size, you can dramatically raise performance. For example, if you can bump your IO size to 128k, you'll double your performance, and depending on your NFS client, you may be able to set rsize, and wsize, to various settings to test performance. There is after all a point of diminishing returns.

And if you have multiple disks, in mirror, or RAID, you'll have to adjust your numbers for that.

Real World Disk Performance is almost never what's printed on the side of the box that the disk came in.