2

I have an aging Centos NFS server with a few different ZFS datasets. I've been migrating the data to a newer NFS server using rsync. The new server is also a Centos ZFS host. My issue is that for one dataset, the referenced size is drastically different between old and new servers, and I can't determine the reason.

I'm not using compression or deduplication. I expected the zfs properties to be the same but I noticed the old host does have the following properties enabled for the dataset (devices, exec and setuid). The only other difference I found is the CentOS and ZFS version:

Old host: zfs 0.6.5, centos 7.3, zfs list shows 2.00T referenced

New host: zfs 0.7.2 centos 7.4, zfs list shows 1.29T referenced

To copy the data, I mounted the old export onto the new server and used the following rsync options: -avhH --delete. Looking at df, the inode count is the same but df does show a big difference in used space (1.3T vs 2T). I used 'find' to build a csv of all files in the dataset along with their size and disk usage. The row count and sum of the size column is identical between the two hosts (the size sum is about 1.29TB) The disk usage is only slightly different between the two hosts and matches closely with 1.3T. I've also tried running lsof on the old server to see if there is some unlinked file that was consuming the difference in capacity but nothing was returned as '(deleted)'... any other ideas or suggestions that might explain the difference?

2
  • Are there any snapshots of the datasets?
    – mmusante
    Jan 31, 2019 at 18:16
  • Yes, all datasets within this zpool have a few snapshots, but I don't think the snapshots are playing a role with the discrepancy. I'm looking only at one dataset's referenced size, seeing that it matches with df, but that the number is much greater than du. AFAIK, the dataset's referenced size isn't affected by the dataset's snapshots (I'm reading the referenced size with zfs list).
    – user
    Feb 4, 2019 at 16:05

1 Answer 1

1

When compression is disabled, ZFS doesn't look for blocks that are filled with zeroes (this happens in zio_compress_data() in the code). So, on your old system, you probably have a bunch of zeroed blocks on disk.

I'm not an rsync expert, but I'd guess that rsync will probably not send zeroed blocks to save time / network bandwidth. On your destination system, this means you'll end up with less space usage because those zeroed-out blocks were never written.

If you want the systems to be exact duplicates of each other for whatever reason, you could use ZFS's built-in replication tools zfs send (on the source system) and zfs receive (on the target). It's also more efficient than rsync because it only has to traverse parts of the filesystem tree that have changes in them, unlike rsync which has to inspect every file's metadata, and read the full contents of the changed files, to see what's different.

2
  • I hadn't considered how truncated files or files containing all zeros would affect the copy. Unfortunately, it doesn't look like that is the cause for my discrepancy. I've confirmed that rsync will 'inflate' truncated files (which could explain why someone might see a larger referenced size on the target dataset when using rsync) and rsync will blindly copy files that contain all zeros. That said, it looks like rsync has an -S option to efficiently handle sparse files but I wasn't using that flag. I also used zfs send and confirmed the target dataset matches the source's referenced size.
    – user
    Jan 31, 2019 at 16:55
  • I've stopped looking at rsync as an action that's introducing the discrepancy as running du locally against the source dataset shows the same discrepancy. I'm now trying to determine what would cause the difference in size when comparing du -sh of the root directory with zfs list's referenced size.
    – user
    Jan 31, 2019 at 17:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .