1

I'm in the middle of generating a 3 TB drive image with GNU ddrescue, and realizing it will be too big for the target drive (since the empty space is filled with 0xAA instead of 0x00, so instead of using a sparse output file, I need actual compression).

The output file is on a btrfs filesystem, which supports per-file compression, but by which method?

  1. https://wiki.archlinux.org/index.php/Btrfs#Compression says:

To apply compression to existing files, use the btrfs filesystem defragment -calg command, where alg is either zlib, lzo or zstd. For example, in order to re-compress the whole file system with zstd, run the following command: 

  # btrfs filesystem defragment -r -v -czstd /

This re-compresses existing files, but seems to be for folders of files, not individual files?

  1. It also says:

Tip: Compression can also be enabled per-file without using the compress mount option; to do so apply chattr +c to the file. When applied to directories, it will cause new files to be automatically compressed as they come.

It's not clear if that will re-compress existing files, though.

  1. https://btrfs.wiki.kernel.org/index.php/Compression#Can_I_force_compression_on_a_file_without_using_the_compress_mount_option.3F says

Setting the compression property on a file using btrfs property set <file> compression <zlib|lzo|zstd> will force compression to be used on that file using the specified algorithm.

This seems like it's for forcing compression of files that would not normally be compressed because they contain data that isn't very compressible?

Which command do I want to convert the entire existing file to (default LZO) compression, and to continue compressing any more data that is written to it in the future, without changing the compression of other files on the volume?

|improve this question|||||
  • So I tried #2 and nothing happened. It just changed the attribute in lsattr. Then I tried #1 without -c and nothing happened. It defragged without paying attention to the compressed bit (finishing immediately). Then I tried #1 with -c on the whole folder, and it compressed the files (as reported by compsize) but didn't change the attributes of them! So only the one I changed has c on it, the rest don't, even though they were compressed by defrag, too. So the compressed attribute and defrag compression have nothing to do with each other? – endolith Jan 24 at 3:52
  • github.com/maharmstone/btrfs/issues/64#issuecomment-341557638 "The flag works the same way as on Linux - it's just a hint for new files. If you want to change the compression type used by a file you have to rewrite." – endolith Jan 24 at 4:09
  • github.com/maharmstone/btrfs/issues/64#issuecomment-408437616 "The correct way of changing the compression type is to mount the filesystem in Linux and then use btrfs filesystem defragment -r -clzo /path/to/folder." – endolith Jan 24 at 4:10
2
# btrfs filesystem defragment -r -v -czstd /

This re-compresses existing files, but seems to be for folders of files, not individual files?

It also works for individual files.

(For directories you need the -r flag, else it'll defragment the metadata of the subvolume the directory belongs to).

chattr +c

It's not clear if that will re-compress existing files, though.

It will not.

btrfs property set <file> compression <zlib|lzo|zstd>

This seems like it's for forcing compression of files that would not normally be compressed because they contain data that isn't very compressible?

It disables the mechanism that detects whether or not a file is compressible and compresses it regardless.
I believe it will only store the file compressed if it's at least somewhat compressible though, more specifically if the compressed data is not larger than the original IIRC.

Also note that that detection mechanism isn't very sophisticated and disables compression on very compressible files too sometimes.

Which command do I want to convert the entire existing file to (default LZO) compression, and to continue compressing any more data that is written to it in the future, without changing the compression of other files on the volume?

btrfs filesystem defragment -clzo /path/to/file

(Note: This will force compression on the file)

|improve this answer|||||
  • If I use filesystem defragment and continue writing to the file, the parts I write after that point will not be compressed, right? filesystem defragment compresses existing parts, but not future parts, while chattr +c compresses future parts, but not existing parts? – endolith Jan 25 at 18:30
  • Afaik defrag will enable forced compression on the file, I don't know if it does that immediately or after the file has been defragmented. Intuition tells me it's the former. – Atemu Jan 27 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.