Why does ZFS on Linux not read from the cache?

Question

I have a test zpool set up with a single, USB-attached 4TB spinning disk which reads at about 40MB/sec.

I have four ~300GB fast SATA2 internal SSDs set up as cache for this disk. I can read about 900MB/sec from this L2ARC.

I am reading a mostly static 400GB dataset from this pool.

The system in question has over 200GB of free memory, which I assume operates at >1500MB/sec for our purposes.

Every 1.0s: zpool iostat -y -v 1 1 ; zpool status -v                                                las2: Tue Oct 16 20:20:07 2018

                                                      capacity     operations     bandwidth
pool                                                alloc   free   read  write   read  write
--------------------------------------------------  -----  -----  -----  -----  -----  -----
storage                                              417G  3.22T    239      2  29.7M   132K
  usb-Seagate_M3_Portable_NM12QHQF-0:0-part1         417G  3.22T    239      2  29.7M   132K
cache                                                   -      -      -      -      -      -
  ata-INTEL_SSDSC2BA400G3_BTTV41450D92400HGN-part3  52.8G   291G     11      0   927K   128K
  ata-INTEL_SSDSC2BA400G3_BTTV423000XA400HGN-part3  52.9G   291G     13      0  1.48M      0
  ata-INTEL_SSDSC2BA400G3_BTTV423406FG400HGN-part3  52.7G   291G      8      0   324K      0
  ata-INTEL_SSDSC2BA400G3_BTTV423406G6400HGN-part3  52.9G   291G      5      0   336K      0
--------------------------------------------------  -----  -----  -----  -----  -----  -----
  pool: storage
 state: ONLINE
  scan: scrub repaired 0B in 1h56m with 0 errors on Sun Oct 14 02:20:56 2018
config:

        NAME                                                STATE     READ WRITE CKSUM
        storage                                             ONLINE       0     0     0
          usb-Seagate_M3_Portable_NM12QHQF-0:0-part1        ONLINE       0     0     0
        cache
          ata-INTEL_SSDSC2BA400G3_BTTV41450D92400HGN-part3  ONLINE       0     0     0
          ata-INTEL_SSDSC2BA400G3_BTTV423000XA400HGN-part3  ONLINE       0     0     0
          ata-INTEL_SSDSC2BA400G3_BTTV423406FG400HGN-part3  ONLINE       0     0     0
          ata-INTEL_SSDSC2BA400G3_BTTV423406G6400HGN-part3  ONLINE       0     0     0

errors: No known data errors

AIUI, reads get cached into the ARC in system memory. When that hits some level of "full", these get evicted onto the L2ARC, 1.1TB of fast SSD.

When I run a tar -c . | pv > /dev/null from one of the mostly-unchanging directories on this pool, I see around 40MB/sec - the speed of reads from the underlying, slow HDD.

The problem is, I see it the second time, too! Sometimes it spikes to 80-100MB/sec, but also sometimes it falls to 10MB/sec. I would expect 99% of the reads to come from memory or the fast L2ARC.

What gives? I would expect to see from-RAM speeds as this box has 256GB of system memory and over 200GB of that is free. Failing that, there's still hundreds of gigabytes of L2ARC unused.

What am I doing wrong? Why is this data not being read from the ARC at extremely high speed (>1GB/sec)? Failing that, why is it not being read from the L2ARC (>500MB/sec)? Why am I seeing 40-80MB/sec on a mostly-unloaded system?

Ubuntu bionic 18.04.1 x86_64 system.

Linux las2 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

zfs-auto-snapshot/now 1.2.4-1 all [installed,local]
zfs-zed/now 0.7.5-1ubuntu16.4 amd64 [installed,local]
zfsutils-linux/now 0.7.5-1ubuntu16.4 amd64 [installed,local]

Answer 1

There are two issues here.

First, your dataset doesn't fit in memory, and gets evicted based on LRU (Least Recently Used). This means that, after you run through the whole dataset, if you start again from the beginning, none of what you're looking for will be in the ARC.

The second problem is that the L2ARC isn't filled quite the way you think. It's filled when data is evicted from the ARC, but the fill rate is throttled. I believe the default limit is 8MB/S, which will have a small impact on your second read pass. To bump this, I believe you should check out the "l2arc_write_max" and "l2arc_write_boost" ZFS parameters.