2

I have a freebsd file server

root@vstorage:~ # freebsd-version 
11.1-RELEASE-p1

Here's some additional info.

root@vstorage:~ # dmesg | grep CPU:
CPU: AMD Phenom(tm) 9750 Quad-Core Processor (2400.05-MHz K8-class CPU)

root@vstorage:~ # dmesg | grep memory
real memory  = 8589934592 (8192 MB)
avail memory = 8001138688 (7630 MB)

Samba client : MacBook Pro - MacOS Sierra - Connected to a 5ghz AC wireless network. File server connected to gigabit router (Netgear Nighthawk x4 r7500)

I just built this file server to replace an ubuntu file server I had previously (which did not have this issue)

I have it serving a NFS share to a hypervisor (working fine), and also serving a Samba share to my Mac.

I installed samba46 from the ports collection. Here is my smb4.conf.

root@vstorage:~ # cat /usr/local/etc/smb4.conf 
[global]
workgroup = WORKGROUP
server string = Samba Server Version %v
netbios name = vstorage
wins support = Yes
security = user
passdb backend = tdbsam
socket options = TCP_NODELAY IPTOS_LOWDELAY
read raw = yes
write raw = yes

[data]
path = /data
valid users = nullmeta
writable  = yes
browsable = yes
read only = no
guest ok = no
public = no
create mask = 0775
directory mask = 0775

The share is a ZFS pool called datathat has a single 3TB western digital red drive.

root@vstorage:~ # zpool status
  pool: data
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    data        ONLINE       0     0     0
      ada1      ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      ada0p3    ONLINE       0     0     0

The Problem:

I can connect to the share just fine. However, if I try to transfer anything to it, it eats all the memory. If I have top running while transferring a 2GB file, I can watch the memory usage go up, it seems to cap out and leave me 470M of free memory.

Here's top after transferring 3 x 2GB files.

    root@vstorage:~ # top

    last pid: 23995;  load averages:  0.30,  0.28,  0.25    up 0+00:40:35  19:56:12
    33 processes:  1 running, 32 sleeping
    CPU:  0.0% user,  0.0% nice,  0.1% system,  0.0% interrupt, 99.9% idle
    Mem: 59M Active, 23M Inact, 6099M Wired, 1503M Free
    ARC: 5674M Total, 17M MFU, 5610M MRU, 35K Anon, 14M Header, 32M Other
         5512M Compressed, 5675M Uncompressed, 1.03:1 Ratio
    Swap: 2048M Total, 2048M Free
PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1122 root          1  20    0   228M 12544K select  1   0:00   0.07% nmbd
23994 root          1  20    0 20164K  2920K CPU3    3   0:00   0.04% top
  338 root          1  20    0  9564K  4844K select  3   0:00   0.02% devd
  411 root          1  20    0 10504K  1860K select  1   0:00   0.01% syslogd
  734 jeff          1  20    0 85232K  5808K select  2   0:00   0.00% sshd
  663 root          1  20    0 20640K  4600K select  1   0:00   0.00% sendmail
  561 root         32  52    0  8332K  2112K rpcsvc  3   0:00   0.00% nfsd
 1134 root          1  20    0   301M 17524K select  3   1:31   0.00% smbd
  558 root          1  20    0 12548K  2760K select  0   0:00   0.00% mountd
  786 root          1  20    0 19664K  3348K pause   3   0:00   0.00% csh
  631 root          1  20    0 64540K 26268K select  0   0:00   0.00% perl
 1128 root          1  20    0   297M 16852K select  2   0:00   0.00% smbd
  543 root          1  20    0 10452K  1920K select  0   0:00   0.00% rpcbind
  560 root          1  23    0 10380K  2956K select  3   0:00   0.00% nfsd
 1130 root          2  20    0   297M 16196K select  3   0:00   0.00% smbd

I have no idea why it is eating the memory. Once its fully utilized, the transfers slow to a crawl. The crazy part is, it will not release the memory either. Disconnecting the share and reloading samba do nothing. The only way to reclaim the memory is to reboot the server, which is quite frustrating since that means I also have to suspend the hypervisor until the storage comes back online.

I thought it was a samba issue, so I reinstalled it, configured socket options, read write/raw, and disabled logging. I don't however think that is my issue, I cannot seem to find the root cause of samba eating the memory when transferring. Any insight would be appreciated.

EDIT: I am beginning to think that this is a ZFS issue rather than a samba issue. I will update if I find out any additional information.

  • ARC: 5674M Total, so ARC cache (it is a thing in ZFS) eats > 5Gb, nothing unusual, it caches it in case someone else will try to download the same file. The question is, why do you want to reclaim this memory? – myaut Sep 17 '17 at 22:17
  • @myaut well I believe it was overloading my server, causing extremely slow transfers since it has no ram left (it was all being used as a cache) I read up on a bunch of stuff and found that setting an arc_max limit in the loader.conf would help. It seems to have, I’m running a few more tests to verify. But when I posted the question I didn’t realize that it was just a cache, and the memory wasn’t actually in use, so I don’t really need to reclaim it after all. – nullmeta Sep 17 '17 at 22:22
  • in Solaris ARC size shrinks during "VM pressure" (i.e. when some processes allocate memory), I believe FreeBSD implementation follows this practice as well. – myaut Sep 17 '17 at 22:27
  • Thank you for the info @myaut . It seems that setting the arc_max limit to 3GB has solved the issue. Transfer speeds stay consistent and quick as they should be. If you are able to elaborate as to why this is, or can recommend another tweak that may work, feel free. I will add an answer to this question after I do some more research. – nullmeta Sep 17 '17 at 22:40
  • @nullmeta You likely don't need even a 3 GB ARC and likely won't notice any peformance loss with something like a 512MB ARC. This wouldn't be the first time I've seen the ZFS ARC creating performance problems. Sure it gets released when you need memory. S-L-O-W-L-Y. The ARC also fragments large pages, so if you have an application that requires large pages your server can completely hang for long periods of time while the VM system coalesces small pages to create the required large pages, large pages that if allowed the ARC will immediately fragment once the application releases them.... – Andrew Henle Sep 19 '17 at 10:12
4

The problem is not with samba at all, but with the ZFS arc cache.

The ZFS arc cache uses physical memory, and since there was no limit set, ZFS would use ALL of the system memory as a cache when transferring files (I was transferring around 200GB at a time with some 30-60 GB files). To fix it, set a limit on the max amount of ram the arc cache is able to use.

As root, use your favorite text editor to edit loader.conf, I will use nano.

root@vstorage:~ # nano /boot/loader.conf

Add the following line to set the arc_max limit.

vfs.zfs.arc_max="1G"

This tells ZFS to use a maximum of 1GB for its arc cache.

After a reboot, the problem is solved, transfers remain at a consistent speed and everything runs smooth.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.