4

Some preamble: I'm taking bitwise copy of disk devices (via dd command) from twin hosts (i.e. with the same virtualized hardware layout and software packages, but with different history of usage). To optimize image size I trailed all empty space on partitions with zeroes (e.g. from /dev/zero). I'm also aware of reserved blocks per partition and temporarily downgraded that value to 0% before trailing.

But I'm curious about discrepancy of the final compressed (by bzip2) images. All hosts have almost the same tar-gziped size of files, but compressed dd images have significant variety (up to 20%). So how could it be? Is there a reason in the filesystem journals data which I was unable to purge? There are over ten partitions on the host and each reported of 128Mb journal size. (I also checked defragmentation, it's all ok: 0 or 1 due to e4defrag tool report)

So, my question is it possible somehow to clean ext3/ext4 filesystem journals? (safely for stored data of course :)

CLARIFICATION
I defenitely asked a question about how to clean (purge/refresh) journals in ext3/ext4 filesystem if possible or maybe I'm mistaken and there is no such feature as reclaiming disk space occupied by filesystem journals, so all solutions are welcome. An intention to ask the question I put as premise into the preamble and the answer to my question would help me to investigate the issue I encountered with.

Improve this question
3
  • 2
    What makes you think the filesystem journal is the major culprit? How about something as mundane as fragmentation causing some storage layouts to compress better than others?
    – user
    Jan 14, 2014 at 12:35
  • Yes I thought about fragmentation too, despite that ext is known as fragmentation resistant I checked fragmentation state and e4defrag showed ok.
    – rook
    Jan 14, 2014 at 12:43
  • Different usage patterns = different data, stored in different locations, so the compression will give different results.
    – psusi
    Jan 14, 2014 at 16:27

2 Answers 2

Reset to default
2

You can purge the journal by either un-mounting, or remounting read-only (arguably a good idea when cloning). With ext4 you can also turn off the journal altogether (tune2fs -O ^has_journal), the .journal magic immutable file will be removed automatically. The journal data will still be on the underlying disk of course, so removing the journal and then zero-filling free space might get the best results.

The comments above hit the nail on the head though, dd sees the bits underneath the filesystem, how they came to be in any particular arrangement depends on all the things that have happened to the filesystem, rather than just the final contents of files. Features such as pre-allocation, delayed allocation, multi-block allocation, nanosecond timestamps and of course the journal itself all contribute to this. Also, there is one potentially random allocation strategy: the Orlov allocator can fall-back to random allocation (see fs/ext4/ialloc.c).

For completeness the secure deletion feature with random scrubbing would also contribute to differences (assuming you deleted your zero-filled ballast files), though that feature is not (yet) mainline.

On many systems the dump and restore commands can be used for a similar cloning method, for various reasons it never quite caught on in Linux.

Improve this answer
0

I found the root cause of the issue, so when I decided to check bitwise copy of each partition instead of whole disk to see how redundant data is distrubuted I realized that /dev/mapper got lv_swap volume (partition) situated on the same disk which was captured and the swap data of course was included into the final image. All image size variety delta was located in that swap partition.. no FS meta magic.. I didn't see it before because my script grabs info from the df command where lv_swap isn't figured.

Anyway, if someone answers the question about purging journals in ext3/ext4 filesystem I'll accept it.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .