In Solaris 11, in a pool named mypool I activated deduplication with

zfs set dedup=on mypool

Then I created two 100 MB file: file1 and its copy file1copy. Only their names are different: their contents is the same.

The mountpoint of mypool is /mypool. If I copy both file1 and file1copy into /mypool, the output of

zpool list mypool

indicates a DEDUP ratio of barely 1.34x. Why??

Then, following this guide I created a filesystem with

zfs create -o dedup=on -o mypool/archive

and then two users' filesystems mypool/archive/a and mypool/archive/b which inherited the deduplication from the parent filesystem.

If now I copy file1 into mypool/archive/a and file1copy into mypool/archive/b there is still a DEDUP ratio of 1.34x. It becomes 2 only if I put the same file, say file1, into both mypool/archive/a and mypool/archive/b. Why??

Can a simple name difference strongly tear down deduplication? Shouldn't deduplication act at block level?

1) Output of zdb -S mypool

Simulated DDT histogram:

bucket              allocated                       referenced          
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    1.48K    189M    189M    189M    1.48K    189M    189M    189M
     2      800    100M    100M   99.9M    1.56K    200M    200M    200M
 Total    2.26K    289M    289M    288M    3.04K    389M    389M    388M

dedup = 1.35, compress = 1.00, copies = 1.00, dedup * compress / copies = 1.35

2) Output of zdb -b mypool

Traversing all blocks to verify nothing leaked ...

    No leaks (block sum matches space maps exactly)

    bp count:            3679
    bp logical:     410970112      avg: 111707
    bp physical:    408433664      avg: 111017     compression:   1.01
    bp allocated:   615948288      avg: 167422     compression:   0.67
    bp deduped:     157286400    ref>1:    800   deduplication:   1.26
    SPA allocated:  458661888     used: 76.47%

In both mypool/archive/a and mypool/archive/b folders there was also another file.

  • What is the output from zdb -S mypool and zdb -b mypool? – Warwick Sep 5 '14 at 0:02
  • I edited my question with that outputs. – BowPark Sep 5 '14 at 9:27
  • The name for the file should not have any impact on deduplication. I strongly suggest you look at the contents of file1 and file1copy and whether they really have the same content (e.g. with sha1sum). – Feuermurmel Dec 1 '16 at 13:16

Not all data is "dedupable". There is an article here that explains this

  • 1
    Thank you for the article. But the blocks of the two files should generate the same hash (they are the same!) and so should be dedupable. Maybe the other files in the filesystem can influence the deduplication? – BowPark Sep 5 '14 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.