10

I packed and compressed a folder to a .tar.gz archive. After unpacking it was nearly twice as big.

du -sh /path/to/old/folder       = 263M
du -sh /path/to/extracted/folder = 420M

I searched a lot and found out that tar is actually causing this issue by adding metadata or doing other weird stuff with it.

I made a diff on 2 files inside the folder, as well as a md5sum. There is absolutely no diff and the checksum is the exact same value. Yet, one file is as twice as big as the original one.

root@server:~# du -sh /path/to/old/folder/subfolder/file.mcapm /path/to/extracted/folder/subfolder/file.mcapm
1.1M    /path/to/old/folder/subfolder/file.mcapm
2.4M    /path/to/extracted/folder/subfolder/file.mcapm
root@server:~# diff /path/to/old/folder/subfolder/file.mcapm /path/to/extracted/folder/subfolder/file.mcapm
root@server:~# 
root@server:~# md5sum /path/to/old/folder/subfolder/file.mcapm
root@server:~# f11787a7dd9dcaa510bb63eeaad3f2ad
root@server:~# md5sum /path/to/extracted/folder/subfolder/file.mcapm
root@server:~# f11787a7dd9dcaa510bb63eeaad3f2ad

I am not searching for different methods, but for a way to reduce the size of those files again to their original size.

How can I achieve that?

  • 1
    Are you saying that is the only file in the archive? – kaylum Jan 19 at 20:06
22

[this answer is assuming GNU tar and GNU cp]

There is absolutely no diff and the checksum is the exact same value. Yet, one file is as twice as big as the original one.

1.1M    /path/to/old/folder/subfolder/file.mcapm
2.4M    /path/to/extracted/folder/subfolder/file.mcapm

That .mcapm file is probably sparse. Use the -S (--sparse) tar option when creating the archive.

Example:

$ dd if=/dev/null seek=100 of=dummy
...
$ mkdir extracted

$ tar -zcf dummy.tgz dummy
$ tar -C extracted -zxf dummy.tgz
$ du -sh dummy extracted/dummy
0       dummy
52K     extracted/dummy

$ tar -S -zcf dummy.tgz dummy
$ tar -C extracted -zxf dummy.tgz
$ du -sh dummy extracted/dummy
0       dummy
0       extracted/dummy

You can also "re-sparse" a file afterwards with cp --sparse=always:

$ dd if=/dev/zero of=junk count=100
...
$ du -sh junk
52K     junk
$ cp --sparse=always junk junk.sparse && mv junk.sparse junk
$ du -sh junk
0       junk
| improve this answer | |
  • MCAPM is a MineCraft Anvil PocketMap file, described as holding "chunks" (segments of a map). Seems eminently likely to be sparse. I worried about the identical checksum, until I remembered null bytes and holes read back the same. – Paul_Pedant Jan 19 at 21:00
  • @schily fwiw, dd count=0 not reading any block is not only the behaviour of GNU dd, also that of {Open,Net,Free}BSD's dd, and is also allowed by the standard: "count=n: ... If n is zero, it is unspecified whether no blocks or all blocks are copied.". That doesn't mean it's right to assume it, as I did. So thanks for the heads-up. – mosvy Jan 20 at 15:00
  • that worked perfectly - thank you! (: – Permittivity Jan 20 at 15:22
8

@mosvy points out that your files were probably sparse. Re-doing the archive + extract with tar --sparse works, or you can make existing files in the filesystem sparse again using
fallocate -d
(from util-linux) to punch holes in in-place.

for f in **/*some*pattern*;do
    fallocate --dig-holes "$f"
done

The man page describes this option as

You can think of this option as doing a cp --sparse and then renaming the destination file to the original, without the need for extra disk space.


Linux supports the fallocate(2) system call which allows cool stuff like this, including closing up or expanding page-sized holes in a file to shorten or grow a file, instead of just turning a range into a hole. It depends on the underlying FS to support each of the various fallocate features separately, and of course sparse files / extents in general.

It also lets you preallocate unwritten extents (like hole but with space reserved on disk), e.g. before a torrent download to avoid fragmentation. That's where the "allocate" in the name comes from.

Other kernels that util-linux can run under may support some or all of this functionality, IDK. If it doesn't work, cp --sparse and rename should work; sparse files in general (seek instead of writing zeros) are well-established and widespread in Unix, dating back much much farther than preallocated extents, punching holes, or especially expanding or collapsing holes between existing data.

| improve this answer | |
  • I don’t get how large “holes” can escape being compressed by gzip. – WGroleau Jan 20 at 18:32
  • @WGroleau: I'm not sure if tar can encode holes using metadata, or if it just lets gzip handle a long run of zeros (which it can compress with very good ratio). But anyway, I'm suggesting using this on the already-extracted (without --sparse) files, not on the tar before gzip or on the .tar.gz (which wouldn't have any 4k runs of zeros). – Peter Cordes Jan 20 at 19:59
  • Oops, I missed that OP said after unpacking. Does tar preserve hard links, or do they become separate copies of the item? – WGroleau Jan 20 at 20:27
  • @WGroleau: I think at least GNU tar can detect hardlinks when archiving (presumably by keeping a hash table of fs:inode of all files its already archived), and make links when extracting. The man page documents a --hard-dereference to not do that, like -h, --dereference does for symlinks. – Peter Cordes Jan 20 at 20:36
  • 1
    @PeterCordes GNU tar is using special format extensions to store sparse files. Unlike what you're suggesting, --sparse has no effect when extracting. You could run fallocate -d on the extracted files instead of cp --sparse, but fallocate only work on some filesystems. The fact that GNU tar is using a special format may create problem with older tar implementations (but bsdtar, the default tar on FreeBSD, and the libarchive it's based on seems to handle it just fine). – mosvy Jan 20 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.