With Buildroot I'm making images for my Embedded Linux Hardware. Mainly I'm trying to speed up the boot sequence (and on the way lower the memory usage), where I've tried many techniques successfully.

What I'd like to do: Recently I've heared about removing duplicated files in a directory (by e.g. replacing those files with symbolic links) and I'd like to apply this method to my rootfs

The surroundings: With Buildroot I can have many different types of rootfs formats (cramfs, cpio, ext2/3/4, etc.), which are created during make as 1 (packed) file (e.g.: rootfs.cpio). Now I don't really know how to:

  • open up the image
  • remove duplicated files (well I know how to remove duplicated files in general)
  • pack the rootfs again - so that I still can use it to flash and execute it on my hardware

Maybe it's not even possible at all. I believe at least when using static libraries, many files can be replaced.

Has somebody an idea?

  • 1
    I compare the md5sum of files to find duplicates. I don't know much about embedded systems, but I know that specialized filesystems such as squashfs do this kind of deduplication automatically. You might want to consider "hard" links instead of symbolic ones in case of picky software that expects to find a real binary. – MattBianco Sep 16 '14 at 7:06
  • I'm surprised that you have duplicated files in the first place. What are they? This smacks of a broken build script. – Gilles 'SO- stop being evil' Sep 16 '14 at 21:24
  • @Gilles well I haven't checked yet, that's what I would like to do. But as mentioned in the top post, I guess at least some libraries can be replaced. I won't know it if I won't test it – user3085931 Sep 17 '14 at 6:04
  • I don't understand why you think some libraries will be duplicated. If you link a library statically, some bits of code will be duplicated, but not files, and not even disk blocks either because the same code will show up at different alignments and possibly with different bits inlined. If you link a library dynamically, the code will only appear once, in the single dynamic library file. – Gilles 'SO- stop being evil' Sep 17 '14 at 6:11
  • Well I get your point, but I don't know what happens if you're including additional Packages into your rootfs. E.g.: if you have multiple crypto programs, i could imagine that everyone of those will come with their own libraries. Or e.g. if you let the user create packages/programs that will be build into the rootfs, so it is really unknown what's build twice and what is not. – user3085931 Sep 17 '14 at 6:17

I can't help with the buildroot specific stuff (other than to say that you'd have to either mount it or boot it before removing duplicates, and then creating a new rootfs from the resulting filesystem), but removing duplicates is quite possible. A naive approach would be to find / -type f -exec sha256sum {} + | sort > checksums.txt, read the resulting file line by line with while read -r checksum path, compare each checksum to the previous, and deleting (or hardlinking) the duplicates. This wouldn't handle weird filenames, for example containing newlines, but that shouldn't be a problem in your case.

| improve this answer | |
  • ... does this also fetch a rootfs compressed to 1 file, like e.g. rootfs.ext4 ? – user3085931 Sep 16 '14 at 10:53
  • I'm afraid I don't understand the question. – l0b0 Sep 16 '14 at 15:06
  • if you're using a build system (like buildroot) then it will create you a rootfs in/as a single file, depending on what you've chosen you'll get e.g. a rootfs.ext4. Now this file can be used to flash to a partition on a SD card with the dd command - and then you have the typical linux file system structure with the home, usr, etc ... directories - so you can configure linux to read the rootfs from the SD-Card partition, which is working pretty fine. – user3085931 Sep 17 '14 at 6:06

I don't see how static libraries enter into the mix. Static libraries are not present on the target root filesystem with Buildroot.

Generally speaking, Buildroot tries to install only what's necessary on the target, and I honestly doubt you will find many duplicated files there.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.