3

EDIT: For some reason only part of my post got saved. I have updated it now to its full content


Although this topic circles around the Raspberry Pi it is applicable to basically all SD cards that have the following setup: a filesystem takes only a fraction of the total size of the card. In short the point of this question is to get a grip on how to write ONLY the FS (or more if multiple partitions are present) to a backup image. Further details can be seen below.

When I started playing around with the Pi after I bought it in December last year I had a couple of corruptions of the SD card and lost its contents. It was due to improperly turning the power off instead of doing a normal shutdown so no biggie here. Since every single time I had to format the card, write the Raspbian image to it, put it in the Pi, boot up the board, expand the filesystem, reinstall and reconfigure everything I started looking for better solutions:

  • resizing the image and the FS on it before writing it to the SD card - this link illustrates perfectly how to increase an image (using dd and /dev/zero as input source) and adjust the ext4 (I think it is EXT4 for the Raspbian (excluding the FAT32 small partition for the booting components)). This solves the issue of having to use the expand filesystem feature of the Raspberry configuration tool which basically eats up the rest of the unformatted space be it 2GB, 4GB, 16GB, 32GB and so on. Of course increasing the image also increases the time dd requires to write it to the SD card however it also allows the next step of customization namely
  • customizing the installed software - using chroot or proot (plus qemu) it is possible (as seen here and here) to install/build whatever you want if you properly mount the Raspbian's FS and chroot in it. This also is an essential step for many builds that require cross-compiling (like Qt for Pi) - you can compile stuff natively on the Pi but it takes a lot of space and a lot of time.

So this is basically the way I've been doing stuff to customize my images. The issue here is the set of procedures involved in the inverse operation - taking the contents of the SD card and storing it on your PC in an optimal way. Compression is of course one of the crucial things you can do however it's not enough. When using dd you can either write the whole SD card (including the empty blocks) to your computer or select the size and number of sectors you want. The second way offers a better usage of storage however it requires carefully calculating stuff. If you cut things short you might end up with a broken FS (oh, have I not seen this many times...).

I'm looking for a solution that is very similar to what Clonezilla offers - you can clone only the blocks that actually have content and dump the rest thus shrinking the FS and being able to store a fully functional FS in a much smaller space. I did that last year when I helped my neighbour save the data on his broken HDD on a MacBook.

Example: if you have a Raspbian Jessie Lite installation, only approx. 1.3GB will be used on your SD card. Now imagine if the card is 32GB. Sure, in time you might want to format and use the remaining space however this is generally not the case (at least for me). So if I use dd blindly I will basically create a 32GB image file. The compression will take care of most of the gigabytes there but writing it back to the SD card at a later point in time to restore some previous state of your Raspbian will be a very, very long process. I can also copy the FS (again with dd). But what if the actual installation (all files) take only 600MB or so on that partition? Why would I want to copy 1.3GB when only 600MB are what I need? And what if I have an SD card with a Raspbian where the "expand filesystem" feature of the configuration tool was used and the whole SD card is formatted even though n% of it is actually being used?

Any ideas how to accomplish this with as less effort and tinkering as possible?

1

Dump/restore the small FAT partition by dd (it that's needed at all), but use e2image -a (from the e2fsprogs package) for the main partition. For restoring to flash, the -c option seems useful (it only overwrites what's really different on the destination).

  • Sounds interesting. Can you elaborate on what you've written? From what I understand two image files will be created - one using dd (for the boot partition) and one using e2image -a (for the Linux FS). If so is it possible to merge the two into a single image or does it involve these two steps in reverse when writing the data back to the SD card? – rbaleksandar Oct 30 '16 at 10:31
  • No, you can't merge the two, they use different formats. You could tar them, but I don't see the point. You could selectively restore either or both. – Ferenc Wágner Oct 31 '16 at 6:41
0

Well, while not directly answering your question, here is what I did for easy restorations

  • dd the SD card contents into a file, after all required packages were installed
  • bzip the file with maximum compression and keep the gzipped data file as backup
  • In case of corruption, restore the backup file after bunzip using dd
  • This is exactly what I'm doing now (not using bzip though) and it's exactly what I'd like to avoid doing. – rbaleksandar Oct 29 '16 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.