2

I just backed up the microSD card from my Raspberry Pi on my PC running a Linux distro using this command:

dd if=/dev/sdx of=file.bin bs=16M

The microSD card is only 3/4 full so I suppose there's a few gigs of null bytes at the end of the tremendous file. I am very sure I don't need that. How can I strip those null bytes from the end efficiently so that I can later restore it with this command?

cat file.bin /dev/zero | dd of=/dev/sdx bs=16M
  • 2
    Stripping off a few bytes affects the physical size of the file, which may cause problems when trying to mount it or write it to a device, and I don't recommend it. I strongly recommend compressing it instead. – Mukesh Sai Kumar Jan 13 '18 at 8:41
  • 4
    I would't be so sure about them all being "null bytes", the card contains at least one filesystem, probably two, and certain structures may be spread over the disk. Why not compress file.bin and use zcat when a restore is needed? – Gerard H. Pille Jan 13 '18 at 8:44
  • I don't think file layout is an issue given that I have provided the way I would restore the image. – iBug Jan 13 '18 at 9:07
  • 2
    The point is that you suppose there are null bytes at the end of the file and you are not sure whether they are actually null bytes or how large those null bytes occupies. It is not impossible that the last byte is not a null byte, depending on various conditions. – Weijun Zhou Jan 13 '18 at 9:20
  • @WeijunZhou Yep, you're right. I'm not sure how much is there nor am I sure if it exists. – iBug Jan 13 '18 at 9:22
5

To create a backup copy of a disk while saving space, use gzip:

gzip </dev/sda >/path/to/sda.gz

When you want to restore the disk from backup, use:

gunzip -c /path/to/sda.gz >/dev/sda

This will likely save much more space than merely stripping trailing NUL bytes.

Removing trailing NUL bytes

If you really want to remove trailing NUL bytes and you have GNU sed, you might try:

sed '$ s/\x00*$//' /dev/sda >/path/to/sda.stripped

This might run into a problem if a large disk's data exceeds some internal limit of sed. While GNU sed has no built-in limit on data size, the GNU sed manual explains that system memory limitations may prevent processing of large files:

GNU sed has no built-in limit on line length; as long as it can malloc() more (virtual) memory, you can feed or construct lines as long as you like.

However, recursion is used to handle subpatterns and indefinite repetition. This means that the available stack space may limit the size of the buffer that can be processed by certain patterns.

| improve this answer | |
  • I finally changed my backup command to this: pv -s $SIZE /dev/sdx | gzip > file.bin.gz where $SIZE is obtained by other means. – iBug Jan 13 '18 at 13:30
  • Anyway, the GNU sed solution is a good one if I later want to work with some small files. – iBug Jan 13 '18 at 15:56
  • Note that if you forget the '<' parameter in gzip it will might try to do some dangerous operation! – Luciano Andress Martini Jan 15 '18 at 11:58
  • Your sed command didn't work reliably for me. For some files it doesn't do anything. – letmaik Apr 21 at 16:37
  • @letmaik First things, first: what operating system and version of sed are you using? Try running sed --version and tell us the result. – John1024 Apr 21 at 19:45
1

You can write a simple tool to solve this problem.

Read the file, find out the last valid byte(not null), then truncate the file.

An example in rust from https://github.com/zqb-all/cut-trailing-bytes:

use std::io;
use std::io::prelude::*;
use std::fs::File;
use std::fs::OpenOptions;
use std::path::PathBuf;
use structopt::StructOpt;
use std::num::ParseIntError;

fn parse_hex(s: &str) -> Result<u8, ParseIntError> {
    u8::from_str_radix(s, 16)
}

#[derive(Debug, StructOpt)]
#[structopt(name = "cut-trailing-bytes", about = "A tool for cut trailing bytes, default cut trailing NULL bytes(0x00 in hex)")]
struct Opt {
    /// File to cut
    #[structopt(parse(from_os_str))]
    file: PathBuf,

    /// For example, pass 'ff' if want to cut 0xff
    #[structopt(short = "c", long = "cut-byte", default_value="0", parse(try_from_str = parse_hex))]
    byte_in_hex: u8,

    /// Check the file but don't real cut it
    #[structopt(short, long = "dry-run")]
    dry_run: bool,
}


fn main() -> io::Result<()> {

    let opt = Opt::from_args();
    let filename = &opt.file;
    let mut f = File::open(filename)?;
    let mut valid_len = 0;
    let mut tmp_len = 0;
    let mut buffer = [0; 4096];

    loop {
        let mut n = f.read(&mut buffer[..])?;
        if n == 0 { break; }
        for byte in buffer.bytes() {
            match byte.unwrap() {
                byte if byte == opt.byte_in_hex => { tmp_len += 1; }
                _ => {
                    valid_len += tmp_len;
                    tmp_len = 0;
                    valid_len += 1;
                }
            }
            n -= 1;
            if n == 0 { break; }
        }
    }
    if !opt.dry_run {
        let f = OpenOptions::new().write(true).open(filename);
        f.unwrap().set_len(valid_len)?;
    }
    println!("cut {} from {} to {}", filename.display(), valid_len + tmp_len, valid_len);

    Ok(())
}
| improve this answer | |
0

I tried John1024's sed command and it worked most of the times but for some large files it didn't trim properly. The following will always work:

python -c "open('file-stripped.bin', 'wb').write(open('file.bin', 'rb').read().rstrip(b'\0'))"

Note that this loads the file into memory first. You can avoid this by writing a proper Python script that processes the file in chunks.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.