EDIT: For some reason only part of my post got saved. I have updated it now to its full content
Although this topic circles around the Raspberry Pi it is applicable to basically all SD cards that have the following setup: a filesystem takes only a fraction of the total size of the card. In short the point of this question is to get a grip on how to write ONLY the FS (or more if multiple partitions are present) to a backup image. Further details can be seen below.
When I started playing around with the Pi after I bought it in December last year I had a couple of corruptions of the SD card and lost its contents. It was due to improperly turning the power off instead of doing a normal shutdown so no biggie here. Since every single time I had to format the card, write the Raspbian image to it, put it in the Pi, boot up the board, expand the filesystem, reinstall and reconfigure everything I started looking for better solutions:
- resizing the image and the FS on it before writing it to the SD card - this link illustrates perfectly how to increase an image (using
/dev/zeroas input source) and adjust the
ext4(I think it is EXT4 for the Raspbian (excluding the FAT32 small partition for the booting components)). This solves the issue of having to use the expand filesystem feature of the Raspberry configuration tool which basically eats up the rest of the unformatted space be it 2GB, 4GB, 16GB, 32GB and so on. Of course increasing the image also increases the time
ddrequires to write it to the SD card however it also allows the next step of customization namely
- customizing the installed software - using
qemu) it is possible (as seen here and here) to install/build whatever you want if you properly mount the Raspbian's FS and
chrootin it. This also is an essential step for many builds that require cross-compiling (like Qt for Pi) - you can compile stuff natively on the Pi but it takes a lot of space and a lot of time.
So this is basically the way I've been doing stuff to customize my images. The issue here is the set of procedures involved in the inverse operation - taking the contents of the SD card and storing it on your PC in an optimal way. Compression is of course one of the crucial things you can do however it's not enough. When using
dd you can either write the whole SD card (including the empty blocks) to your computer or select the size and number of sectors you want. The second way offers a better usage of storage however it requires carefully calculating stuff. If you cut things short you might end up with a broken FS (oh, have I not seen this many times...).
I'm looking for a solution that is very similar to what Clonezilla offers - you can clone only the blocks that actually have content and dump the rest thus shrinking the FS and being able to store a fully functional FS in a much smaller space. I did that last year when I helped my neighbour save the data on his broken HDD on a MacBook.
Example: if you have a Raspbian Jessie Lite installation, only approx. 1.3GB will be used on your SD card. Now imagine if the card is 32GB. Sure, in time you might want to format and use the remaining space however this is generally not the case (at least for me). So if I use
dd blindly I will basically create a 32GB image file. The compression will take care of most of the gigabytes there but writing it back to the SD card at a later point in time to restore some previous state of your Raspbian will be a very, very long process. I can also copy the FS (again with
dd). But what if the actual installation (all files) take only 600MB or so on that partition? Why would I want to copy 1.3GB when only 600MB are what I need? And what if I have an SD card with a Raspbian where the "expand filesystem" feature of the configuration tool was used and the whole SD card is formatted even though
n% of it is actually being used?
Any ideas how to accomplish this with as less effort and tinkering as possible?