0

I hear that ZFS has deduplication, so you can save the same file in multiple places but only use ~1 copy's worth of space. I've noticed I tend to end up with music/images/etc. in multiple places across my hard drive over time, and thought it'd be good to make a ZFS partition and move my media to that. I'm having a hard time finding how to do that, or if it's possible, though. Most tutorials say like, "take 3 disks and RAID them together with ZFS", but I'm on a laptop with a single disk, and a preexisting OS I don't intend to erase. I see comments about ZFS partitions (https://serverfault.com/questions/628632/should-i-create-zfs-zpools-with-whole-disks-or-partitions), but I have yet to find anything that explains how to add a single ZFS partition to an existing disk. Can such a thing be done? (My disk's partition table type is msdos, btw, and I'm running Ubuntu.)

3
  • 2
    For a simple application like you describe, data dedup is way overkill, and requires a significant amount of RAM to do the deduplication. Since you seek to "move your media" to the new storage, have you tried simply grooming your existing media storage to get rid of the duplicate files, and storing all your media in one single directory heirarchy? It sounds like your root problem is from not being consistent in where you store your media, and you're proposing moving it to a consistent location. If you'd simply adopt that consistency, your need for a RAM-intensive ZFS solution would go away.
    – Jim L.
    Oct 6, 2021 at 18:10
  • @JimL. Well, except that then I'd need to MAINTAIN that structure, and actively think about not creating duplicates. Much better if the computer will do it for me. Plus, with deduplication I could intentionally file things in multiple places (organized by author as well as by album), if I felt like it.
    – Erhannis
    Oct 7, 2021 at 2:03
  • 1
    Btrfs also has deduplication
    – phuclv
    Oct 8, 2021 at 3:30

1 Answer 1

3

If your goal is to maximize your media organization while maintaining the space saved, that can be accomplished by laying out a directory tree for your media library, and then making judicious use of cp commands (which duplicate data), and ln commands (which create hard or soft [if -s is used] links) to access or duplicate the reference data, even if it's just for the convenience of having the same file available in a different directory and under a different name. Still the same file, just a different hard or soft link to it.

At the manual level, and in very a simplistic view, ln de-dupes, while cp duplicates.

Adding ZFS offers the immediate benefit that you don't have to remember to use ln when you want to use cp. There are lots of use cases such as when media is intentionally sectioned, modified, edited, sequenced, etc. where storing separate "clips" of media can benefit from ZFS de-duping.

The cost for ZFS is that deduplication is rather RAM-intensive. The kernel has to store the checksum or hash of each block on disk and access that array in real time as blocks are written back to the pool. So as your disk fills up (more ZFS blocks get written to), more of your RAM gets filled up as the kernel caches the hash of each block's data. Depending on how much RAM your laptop has, and how much disk, this may or may not be a problem, but it's good to be aware that the primary cost of deduplication is having enough RAM to store the entire disk's hash array. It can be done with less RAM, but write access especially will suffer when the kernel tries to swap the hash array in and out of RAM in pieces on a system without enough RAM to hold the entire hash array at once.

But with all that said, one way to experiment with a small ZFS pool could be:

Create a virtual block device

I'm using FreeBSD commands here, but the steps will be similar on Ubuntu. My example block device will be 10G in size, but you can use a larger size if you have the available space.

  • create a file ztest.dat 10G in size:

    # truncate -s 10G ztest.dat

  • And create a virtual block device that is backed by that file:

    # mdconfig -f ztest.dat

    md0

Our virtual device then is /dev/md0.

Create a simple ZPOOL and enable data deduplication

# zpool create ztest md0
# zfs set dedup=on ztest
# zpool status ztest
  pool: ztest
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    ztest       ONLINE       0     0     0
      md0       ONLINE       0     0     0

errors: No known data errors

That will mount the ztest filesystem at /ztest. Let's also craft a simple shell function that will quickly show some basic stats of the /ztest directory, filesystem and ZFS pool:

zlist() {
  zfs list ztest
  echo
  zfs get all ztest | egrep 'used|refer'
  echo
  zpool get dedupratio ztest
}

With these tools then, you can create your desired media heirarchy under /ztest and begin to copy some sample media files into the /ztest filesystem:

# ls -l /ztest; echo; zlist
total 0

NAME    USED  AVAIL     REFER  MOUNTPOINT
ztest   114K  9.20G       24K  /ztest

ztest  used                  114K                   -
ztest  referenced            24K                    -
ztest  usedbysnapshots       0B                     -
ztest  usedbydataset         24K                    -
ztest  usedbychildren        90K                    -
ztest  usedbyrefreservation  0B                     -
ztest  logicalused           42K                    -
ztest  logicalreferenced     12K                    -

NAME   PROPERTY    VALUE  SOURCE
ztest  dedupratio  1.00x  -

A dedupratio of 1.00 means that no deduplication has been done; the logical size of the data in the filesystem is 1.00 times the size of the physical data stored in the pool.

But we haven't even copied anything in there yet! Let's do so now:

# cp -vp ~jim/my-fave-dvd.iso /ztest/copy1.iso
/home/jim/my-fave-dvd.iso -> /ztest/copy1.iso

And examine the results:

# ls -l /ztest; echo; zlist
total 3271173
-rw-r--r--  1 jim  jim  3347775488 Jul  1  2014 copy1.iso

NAME    USED  AVAIL     REFER  MOUNTPOINT
ztest  3.13G  6.08G     3.12G  /ztest

ztest  used                  3.13G                  -
ztest  referenced            3.12G                  -
ztest  usedbysnapshots       0B                     -
ztest  usedbydataset         3.12G                  -
ztest  usedbychildren        7.18M                  -
ztest  usedbyrefreservation  0B                     -
ztest  logicalused           3.12G                  -
ztest  logicalreferenced     3.12G                  -

NAME   PROPERTY    VALUE  SOURCE
ztest  dedupratio  1.00x  -

The dedup ratio is still 1.00 because we've only stored one copy of our first media file. Let's create a duplicate copy:

# cp -vp /ztest/copy1.iso /ztest/copy2.iso
/ztest/copy1.iso -> /ztest/copy2.iso
# ls -l /ztest; echo; zlist
total 6542345
-rw-r--r--  1 jim  jim  3347775488 Jul  1  2014 copy1.iso
-rw-r--r--  1 jim  jim  3347775488 Jul  1  2014 copy2.iso

NAME    USED  AVAIL     REFER  MOUNTPOINT
ztest  6.25G  6.07G     6.24G  /ztest

ztest  used                  6.25G                  -
ztest  referenced            6.24G                  -
ztest  usedbysnapshots       0B                     -
ztest  usedbydataset         6.24G                  -
ztest  usedbychildren        10.6M                  -
ztest  usedbyrefreservation  0B                     -
ztest  logicalused           6.24G                  -
ztest  logicalreferenced     6.24G                  -

NAME   PROPERTY    VALUE  SOURCE
ztest  dedupratio  2.00x  -

Now the dedup ratio is 2.00, because the logical data referenced by the file system is 2.00 times larger than the data actually stored in the pool. But note that the output from zfs list shows that although the USED value has gone up to 6.25G, the AVAIL value has barely changed, at 6.07G. Let's create some more duplicate copies:

# cp -vp /ztest/copy1.iso /ztest/copy3.iso
/ztest/copy1.iso -> /ztest/copy3.iso
# cp -vp /ztest/copy1.iso /ztest/copy4.iso
/ztest/copy1.iso -> /ztest/copy4.iso
# ls -lh /ztest; echo; zlist
total 13084690
-rw-r--r--  1 jim  jim   3.1G Jul  1  2014 copy1.iso
-rw-r--r--  1 jim  jim   3.1G Jul  1  2014 copy2.iso
-rw-r--r--  1 jim  jim   3.1G Jul  1  2014 copy3.iso
-rw-r--r--  1 jim  jim   3.1G Jul  1  2014 copy4.iso

NAME    USED  AVAIL     REFER  MOUNTPOINT
ztest  12.5G  6.07G     12.5G  /ztest

ztest  used                  12.5G                  -
ztest  referenced            12.5G                  -
ztest  usedbysnapshots       0B                     -
ztest  usedbydataset         12.5G                  -
ztest  usedbychildren        10.6M                  -
ztest  usedbyrefreservation  0B                     -
ztest  logicalused           12.5G                  -
ztest  logicalreferenced     12.5G                  -

NAME   PROPERTY    VALUE  SOURCE
ztest  dedupratio  4.00x  -

So now, du thinks we have 12G of files stored:

# du -h /ztest
 12G    /ztest

Yet zpool shows we've allocated only 3.14G of data in the pool, and that the pool has 6.36G of unallocated space remaining.

# zpool list ztest
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ztest  9.50G  3.14G  6.36G        -         -     0%    33%  4.00x    ONLINE  -

Equivalent use of ln without deduplication

On the other hand, a file system that lacks deduplication can still be used wisely to accomplish the same end, and perhaps with less cost in complexity, RAM requirements, and speed.

# zpool destroy ztest
# zpool create ztest md0
# cp -vp ~jim/my-fave-dvd.iso /ztest/copy1.iso
/home/jim/my-fave-dvd.iso -> /ztest/copy1.iso
# ln /ztest/copy1.iso /ztest/copy2.iso 
# ln /ztest/copy1.iso /ztest/copy3.iso 
# ln -s /ztest/copy1.iso /ztest/copy4.iso 

So now we still have one file, accessible from four different locations (the master copy copy1.iso, and the copies copy2-copy4.iso).

du shows we have 3.1G of files stored:

# du -h /ztest
3.1G    /ztest

ls shows us that many of the inode numbers are the same (hard links), and one file is a symbolic link. Use of ln allows us to place the same file in numerous locations without incurring any additional storage overhead.

# ls -lih /ztest
total 9813518
  2 -rw-r--r--  3 jim   jim     3.1G Jul  1  2014 copy1.iso
  2 -rw-r--r--  3 jim   jim     3.1G Jul  1  2014 copy2.iso
  2 -rw-r--r--  3 jim   jim     3.1G Jul  1  2014 copy3.iso
128 lrwxr-xr-x  1 root  wheel    16B Oct  7 13:25 copy4.iso -> /ztest/copy1.iso

The zfs filesystem shows 3.1G used, 6.1G available:

# zlist
NAME    USED  AVAIL     REFER  MOUNTPOINT
ztest  3.12G  6.08G     3.12G  /ztest

ztest  used                  3.12G                  -
ztest  referenced            3.12G                  -
ztest  usedbysnapshots       0B                     -
ztest  usedbydataset         3.12G                  -
ztest  usedbychildren        189K                   -
ztest  usedbyrefreservation  0B                     -
ztest  logicalused           3.12G                  -
ztest  logicalreferenced     3.12G                  -

NAME   PROPERTY    VALUE  SOURCE
ztest  dedupratio  1.00x  -

And zpool likewise shows that we've allocated 3.12G and have 6.38G available.

# zpool list ztest
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ztest  9.50G  3.12G  6.38G        -         -     0%    32%  1.00x    ONLINE  -

I hope this example will encourage you to develop your own test framework to experiment with ZFS deduplication and see whether the cost/benefit ratio pencils out for you.

Dodging the question

This experimental framework of a ZFS pool on a file-backed virtual device is not a long-term solution for adding a ZFS pool to an existing system. But it can help you determine whether your current hardware can provide the resources necessary, and whether the data and data-handling practices in your specific use case can leverage the benefits that ZFS provides.

If you find that adding a ZFS pool is a desireable goal, then you have the usual prerequisites to think about whenever one is messing about with new disk partitions and filesystems:

  • Have you backed up recently? Do so now!
  • Do you have an empty partition already on your disk? Very handy.
  • No empty partition available? Then do you have enough unused space in your current filesystem that you could back it up, re-size to a smaller partition, and restore your data, leaving the newly-vacated space available for a new partition dedicated to ZFS?
  • Can you buy a new, larger drive and create judiciously-sized partitions there to hold your current filesystem, plus the new ZFS partition?

Once you have:

  • backed up your existing filesystems
  • a dedicated partition to use for ZFS (let's call it /dev/sda3)
  • decided that you want your ZFS pool to be mounted at /tank

You can add a ZFS pool to your existing system with:

# zpool create tank /dev/sda3

If desired, you can enable deduplication with:

# zfs set dedup=on tank

Bear in mind that this will be a non-redundant pool: if your drive fails, your data is gone. zpool scrub operations will be able to detect data corruption, but will not be able to correct any errors found.

Enjoy!

4
  • +1 nice answer. one thing that might be worth adding is that some people do de-duplication with ln and md5sum to find duplicate files. this is very disk IO intensive, though, as it needs to read each file to calculate its checksum. there's also the issue of choosing which of the dupes to keep (and which to replace with a link - and it's important to keep any file that already has multiple links!). The disk IO could be minimised by keeping md5sum's output in a file and only running md5sum on files not already listed in it.
    – cas
    Oct 8, 2021 at 2:03
  • As for ZFS de-duplication - IMO it's not worth the bother. Disk space is much cheaper than RAM. It's easier to just add another drive, or replace a small drive with a larger one. ZFS' built-in compression, though, that's definitely worthwhile if your files are compressible (i.e. text files and many binaries, but not media files like audio or video which are already compressed).
    – cas
    Oct 8, 2021 at 2:06
  • One of the few good use-cases for zfs de-dupe is if you have hundreds or thousands of virtual machine images (in zvols, or in qcow2 or raw files) that are mostly the same. That can save a lot of disk space and can even improve disk performance because the de-duped VM images are probably small enough to fit in cache.
    – cas
    Oct 8, 2021 at 2:11
  • Thanks for answering my question! And a very thorough answer, too. Re: RAM usage, I expect to read from the pool far more often than write to it - it sounds like the hashes are only needed during writing, is that correct? And is there a way to tell ZFS to only load the hashes on a write, or leave them on disk, period, or restrict the RAM it uses for the task? Also - any estimate how much RAM the hashes use per GB disk space?
    – Erhannis
    Oct 12, 2021 at 1:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.