0

I have a strange issue I haven't seen before. I have a ZFS volume which reports it's about 1.7Gb in size, and so does a snapshot. If I then try to do a zfs send for backup purposes, I get a HUGE file - My automated backup, gzipped, results in a 12Gb file, and when I did a test just now (without gzip) I aborted after the file grew to 66Gb - which indicates there is a LOT of repeated data. What's going on here? Fragmentation? If so, what to do?

# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd    476G  64.8G   411G         -    60%    13%  1.00x  ONLINE  -

Volume:

# zfs list
NAME                                                                                  USED  AVAIL  REFER  MOUNTPOINT
lxd                                                                                  64.8G   396G    24K  none
...
lxd/containers/cdinspector                                                            993M  1.32G  1.68G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/cdinspector

Snapshot:

# zfs list -t snapshot
NAME                                                                                           USED  AVAIL  REFER  MOUNTPOINT
lxd/containers/cdinspector@test                                                               1.39M      -  1.68G  -

list -r

# zfs list -r lxd/containers/cdinspector
NAME                         USED  AVAIL  REFER  MOUNTPOINT
lxd/containers/cdinspector  1.05G  1.31G  1.69G  /var/snap/lxd/common/lxd/storage-pools/lxd/containers/cdinspector

Command used to stream the volume:

# zfs send lxd/containers/cdinspector@test | /usr/bin/mbuffer -m 500M > /backup/test
  • 1
    Can you please edit your post to include the full output of zfs list -r lxd/containers/cdinspector? It looks like your output is just zfs list -ro name. – Jim L. Aug 15 '19 at 22:55
  • It's definitely not fragmentation. As zfs send only sends the blocks that are in use by the snapshot. the most likely answer is that lxd/containers/cdinspector is bigger than you thought it was and contains a lot of highly-compressible data. (note: "highly-compressible" is not the same as "repeated"). – cas Aug 16 '19 at 3:07
  • @JimL. I have added the requested information, but it doesn't tell us much :/ – Arni J Aug 17 '19 at 13:17
  • @cas I realize it's not fragmentation now that I think about it, that makes no sense. But yeah it's as if it's referencing a lot more data, somehow? But in any case, I can't pin down an explanation for this ... – Arni J Aug 17 '19 at 13:18
  • try zfs list -r -t all -o space lxd/containers/cdinspector - by default, zfs list -r only displays filesystems, not snapshots. – cas Aug 17 '19 at 16:13
0

The issue turned out to be rooted in ZFS compression. The client in question had a zombie vim process running in his container which was continuously writing repeated data to a swp file.

ZFS compression sure does work! As it turns out, the logical space consumed was about 2.4Tb and with an impressive compression ratio of 1700x then ZFS reduced that to about 1.7Gb physical.

For an article which describes the issue exactly, see:

https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSCompressionAndQuotas

We are now also considering disabling zfs compression, as this is an avenue for a potential DOS attack on us due to the way we've designed things (albeit a very slow, rather obscure attack which requires us to not pay attention).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.