2

Although I am using ZFS since quite a while, I am still failing to understand some aspects of it from time to time. Currently I am trying to understand how ZFS snapshots take up space on the disk and why that space is much smaller then I would expect.

My problem is best explained by an example. I have a VM running from a ZVOL with no compression (compression=off). These are the snapshots of that volume:

root@server01 ~ # zfs list -r -t all -o name,type,available,used,referenced,usedbyrefreservation,usedbydataset,usedbychildren,usedbysnapshots,volsize,refreservation,reservation rpool01/vm-server01
NAME                                       TYPE      AVAIL   USED  REFER  USEDREFRESERV  USEDDS  USEDCHILD  USEDSNAP  VOLSIZE  REFRESERV  RESERV
rpool01/vm-server01                        volume    1.60T  1.97T  1.01T             0B   1.01T         0B      985G    1.50T       none    none
rpool01/vm-server01@Y-2020-05-27-11-35-15  snapshot      -  1.06G  1.00T              -       -          -         -       1T          -       -
rpool01/vm-server01@T-2020-06-02-11-41-15  snapshot      -  1.04G  1.00T              -       -          -         -       1T          -       -
rpool01/vm-server01@Y-2021-04-24-05-36-24  snapshot      -  1.66G  1.00T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@M-2021-04-24-21-22-30  snapshot      -  3.78G  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@T-2021-04-25-14-27-15  snapshot      -     0B  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@T-2021-04-25-14-27-30  snapshot      -     0B  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@W-2021-04-25-21-55-43  snapshot      -   555M  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@D-2021-04-27-17-49-00  snapshot      -  1.52G  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@D-2021-04-29-08-48-16  snapshot      -  1.06G  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@D-2021-05-03-09-42-01  snapshot      -  1.08G  1.01T              -       -          -         -    1.50T          -       -
rpool01/vm-server01@D-2021-05-04-12-12-01  snapshot      -  45.3M  1.01T              -       -          -         -    1.50T          -       -

So far, so good. The worrying thing is:

For example, after having taken the next-to-last snapshot, I have copied about 18 GB of new data to the running VM. However, that snapshot's USED size is reported as 1.08 GB. In my understanding, when a snapshot has been taken, it is read-only and thus actually can't increase. But of course, the file system needs space to record the changes that happen to the dataset / ZVOL afterwards, and that's what it is reported as a snapshot's USED size (please correct me if I am wrong).

As a second, more extreme example, that VM ran about seven hours after I had taken snapshot @T-2021-04-25-14-27-30. I am absolutely sure that quite a few GB of data have been changed in the VM during that time. But that snapshot's USED size even is reported to be 0.

I have found a weird way to see how much data has actually changed after a snapshot has been taken: We can "simulate" sending the snapshot by something like the following command line (command on the first line, output on the following two lines):

root@server01 ~ # zfs send -v -n -R -i rpool01/vm-server01@D-2021-05-03-09-42-01 rpool01/vm-server01@D-2021-05-04-12-12-01
send from @D-2021-05-03-09-42-01 to rpool01/vm-server01@D-2021-05-04-12-12-01 estimated size is 18.6G
total estimated size is 18.6G

(-n tells zfs to not do anything, but report what it would do; -v means verbose; -i means incremental; for -R, please have a look into man zfs, it's too long to explain here)

Here we see that incrementally sending the last snapshot based on the next-to-last snapshot would take approximately 18 GB, which nearly exactly is the amount of data which has been changed in or added to the VM, respectively, after @D-2021-05-03-09-42-01 had been taken, before D-2021-05-04-12-12-01 had been taken. In other words, ZFS knows how much data has been altered between the next-to-last and the last snapshot, but still for the next-to-last snapshot shows a USED value of 1.08 GB instead of about 18 GB.

Could anybody please give an explanation?

P.S. I already have read several articles about how to interpret zfs list's size values, and I have understood that it can get quite hard to understand when reservations, refreservations, nested datasets, clones, snapshots and the like come into play, but the situation here is quite simple, isn't it? Anyway, I didn't see a hint about the difference between the expected and the reported size of snapshots yet.

7
  • 1
    was the 18GB of data highly compressible? plain text files? It's not unusual to get 90% or better compression with highly repetitive text like log files.
    – cas
    Commented May 4, 2021 at 14:41
  • Thank you very much. I forgot to mention that compression is turned off for that ZVOL. I'll ad this to the question. Anyways, in this case, it was 18 GB / about 15.000 messages of email data which has been copied into a IMAP server's mail store which is running in the VM. However, (in other environments) I have observed the same phenomenon after having copied 50 GB of non-compressible data (JPEG and AVC).
    – Binarus
    Commented May 4, 2021 at 14:48
  • not an answer to your question, but why disable compression for that zvol? you'd likely get a massive I/O performance boost if the mail store was compressed. and use a lot less space on your pool. The're really no good reason not to use compression unless the dataset/zvol is going to mostly store data that's already compressed.
    – cas
    Commented May 4, 2021 at 14:53
  • Thanks again. Compression is disabled because the file system in the VM will be encrypted later (by means of the software in the VM itself, in this case Veracrypt / Windows). Therefore, sadly, data won't be compressible from the perspective of ZFS, and turning on compression on that ZVOL would worsen things instead of improving them.
    – Binarus
    Commented May 4, 2021 at 14:56
  • Ok, that may be your answer then. Veracrypt may be (probably is) compressing the data as it's encrypting it - it's fairly common for encryption SW to do this. Check the Veracrypt docs to find out. If it is, then ZFS probably only sees the post-encrypted+compressed data when it's written to the zvol (depends on what you mean by "encrypted later").
    – cas
    Commented May 4, 2021 at 15:02

1 Answer 1

2

I now can answer my own question. Since I've got no reply for a few days here, I've asked the question on the ZFS on Linux mailing list as well, and two members named "Greek" and "Stefan Ring" have explained the subject to me. The thread is here.

To summarize:

The USED size of a snapshot (in contrast to my first understanding) does not reflect the amount of data which was written to the ZVOL since that snapshot has been taken, before the next snapshot has been taken.

Instead, it is the amount of data which is unique to that snapshot. In other words, it is the amount of disk space which would be freed if you would delete that snapshot.

Now it is clear why this never is the amount of data you have added to the ZVOL before the next snapshot has been taken. To come back to my example: I have added about 18 GB to the ZVOL after the next-to-last snapshot has been taken.

However, if I would delete that snapshot, those 18 GB of data of course would still be in the ZVOL. Therefore, that space would not be freed by deleting that snapshot.

Instead, only about 1 GB (the USED size) would be freed; only that space is unique to the snapshot and is probably occupied by a mixture of metadata to manage the snapshot and payload data which has been deleted or changed (but not added) after that snapshot.

If you want to see the amount of data which has been written to a snapshot since it has been taken and before the next snapshot has been taken, use the WRITTEN property.

The difference between USED and WRITTEN is clearly documented in the zfs manual. Although I have read that manual more than one time, I obviously have missed the WRITTEN property.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .