3

I wish to dump the raw content of a SD card into a file for inspection. Most parts of it are zeroes. Learnt from this SuperUser answer I could use pv to show the progress of od and hexdump. Both ETA are 1.5 hours.

# pv /dev/sdd | od -x --endian=big > sdd_file
... ... ... [>                                     ] ... ETA 1:34:42

and

# pv /dev/sdd | hexdump -C > sdd_file
... ... ... [>                                     ] ... ETA 1:35:01

However xxd would need 11 hours.

# pv /dev/sdd | xxd -a -u > sdd_file
... ... ... [>                                     ] ... ETA 10:48:53

I prefer xxd mostly because of the -revert possibility. But xxd would take way too long to process a disk. How to format hexdump (or od) to produce the same file format as xxd, so that the file is possible to be -reverted by xxd?

2 Answers 2

1

Some said xxd -r also accepts hexdump output as input. But I tested it is not the case. In addition if the file serves as a backup as well, it is better to be in a more guaranteed (i.e. the exact original) format.

Thanks to the inspiration from this answer I learnt about formatting the output of hexdump.

TL;DR - The solution

pv /dev/sdd | hexdump -e '"%08.8_ax: "' -e '2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " "' -e '" " 16/1 "%_p" "\n"' > sdd_file

And with a few fixes.

Please note. - If the disk is mostly zeroed, these methods produce a very small file, similar to compression. Otherwise, the output file is around 4 times of the disk size. Be aware if your drive has enough free spaces.

To test it

Prepare the file.

# echo '- - - - Create a really large file of zeroes - - - -'
# dd bs=1100000000 count=4 if=/dev/zero of=test
4+0 records in
4+0 records out
4400000000 bytes (4.4 GB, 4.1 GiB) copied, 8.71123 s, 505 MB/s

# echo '- - - - Overwrite it with some letters in the beginning (without \n) - - - -'
# echo -n "ABCD xyz" > letters
# dd if=letters of=test conv=notrunc

# echo '- - - - Append some letters in the end (with \n) - - - -'
# echo "ABCD xyz" >> test

What xxd -a -u outputs.

# pv test | xxd -a -u > test_xxd
4.10GiB 0:05:39 [12.3MiB/s] [====================================================>] 100%

# cat test_xxd
00000000: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
10642ac00: 4142 4344 2078 797A 0A                   ABCD xyz.

How hexdump ... (my solution) mimics the output.

# pv test | hexdump -e '"%08.8_ax: "' -e '2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " "' -e '" " 16/1 "%_p" "\n"' > test_hexdump
4.10GiB 0:00:29 [ 144MiB/s] [====================================================>] 100%

# cat test_hexdump
00000000: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
10642ac00: 4142 4344 2078 797A 0A                   ABCD xyz.

Compare them.

# diff -s test_xxd test_hexdump
Files test_xxd and test_hexdump are identical

Explanation

The format syntax is well written in the hexdump man page.

   A format string contains any number of format units, separated by
   whitespace. A format unit contains up to three items: an
   iteration count, a byte count, and a format.

   The iteration count is an optional positive integer, which
   defaults to one. Each format is applied iteration count times.

   The byte count is an optional positive integer. If specified it
   defines the number of bytes to be interpreted by each iteration
   of the format.

   If an iteration count and/or a byte count is specified, a single
   slash must be placed after the iteration count and/or before the
   byte count to disambiguate them. Any whitespace before or after
   the slash is ignored.

My solution contains three format strings, all followed by the -e option.

-e '"%08.8_ax: "'

As mentioned in the man page, a format unit is { { iteration_count:1 / } byte_count } format. In this case both the iteration count and the byte count are omitted. _a prints the offset byte in hex format. %08.8 means it occupies 8 characters and preceded with zeroes. (In fact just %08_ax would do.)

-e '2/1 "%02X" " " ... ... ... ... ... '

The second format string is a repetition of the format units 2/1 "%02X" " " for 8 times; there are 8 columns. 2/1 means to consume 1 byte for 2 times. This makes sure the output hex is in big endian format. If we consume 2 bytes and convert it into hex, the byte on right will be regarded as the most significant byte (#1). The output will appear swapped, in contrast to our numbering system as well as our intuitive perception.

The format "%02X" means to display it as uppercase HEX at 2 characters width with left padding 0.

" " is in fact another format unit with omitted iteration count and byte count. It adds a space in between every 2 bytes.

Then this group of format units is repeated for 8 times, printing the 8 columns.

#1 - hexdump by default uses the CPU's endianness (ref), which x86 / x64 CPUs use little endian with 16-bit size word.

-e '" " 16/1 "%_p" "\n"'

Lastly, the third format string simply starts with a space " ". Then it is followed by a format unit of 16/1 "%_p". Again, it consumes 1 byte at a time, and iterates for 16 times. %_p outputs a character in the default character set, as hexdump -C and xxd normally does by default. The third format unit of this format string \n simply outputs a new line character.

When the odds are not in our favours

There are two things we need to fix.

TL;DR - (1) Add back the final line, if necessary. (2) Add back the repeating non-null lines.

Observe the differences between their line skipping algorithms.

# xxd -a -u test
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000030: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00000070: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000080: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000090: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

# hexdump -C test
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000020  41 42 43 44 20 78 79 7a  00 00 00 00 00 00 00 00  |ABCD xyz........|
*
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000070  41 42 43 44 20 78 79 7a  00 00 00 00 00 00 00 00  |ABCD xyz........|
*
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000d0

xxd

  1. xxd always print the final line. The final line is excluded in the repeating lines consideration.
  2. For three or more repeating null lines, xxd skips the consecutive lines.
  3. While restore, xxd restores asterisk (*) back to null lines only. xxd will not consider asterisk as repeated non-null lines. It is because xxd does not skip non-null lines in the first place.
  4. Even though in rule (2), if there are two consecutive null lines, xxd will not skip any one of them. However while restore, xxd accepts an askerisk to be restore into one null line only. xxd handles this one well. (We will prove this later.)

hexdump

  1. hexdump always print one extra line for the file byte count when using the default format. This is why hexdump could skip the the final line if it is a repeating line.
  2. For two or more repeating null or non-null lines, hexdump skips the consecutive lines.

Demo of our solution.

# pv test | hexdump -e '"%08.8_ax: "' -e '2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " " 2/1 "%02X" " "' -e '" " 16/1 "%_p" "\n"' > output
 208 B 0:00:00 [1.00MiB/s] [==================================>] 100%
# cat output
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00000020: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
*
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00000070: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
*
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*  

Two fixes.

  1. If the final line is an asterisk, which means the final line is skipped by hexdump. We have to calculate the byte offset, which is the file (or disk) size minus 16 byte (i.e. 0x10). Either add back the final line, or replace the asterisk by the final line.
  2. If the line before an asterisk is not a null line, which means hexdump skipped the non-null lines. We have to add back these non-null repeating lines.
  3. Please note while adding the final line, it may be a null line, or a non-null line, is needed.

To find the disk size, we may use lsblk -b.

After the fix.

# vi output
# cat output
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00000020: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000030: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00000070: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000080: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
00000090: 4142 4344 2078 797A 0000 0000 0000 0000  ABCD xyz........
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
000000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Restore and check.

# xxd -r output restore

# diff -s test restore
Files test and restore are identical

Please note. At line 00000010: and line 000000b0, the single null line replaced by asterisk, is well handled by xxd -revert.

-1

You can simply use below things

  1. Get hexdump in required format

    hexdump -Cv <binary_file.bin> > binary_file.txt

-C will print data byte by byte

-v will not use * for repeated content

  1. Now you need to remove First & Last few char from binary_file.txt for each line. You can use sed command here.

  2. Now use xxd command

    xxd -r -p binary_file.txt > recover_binary_file.bin

  3. You can check md5sum of both files. It should match

    md5sum binary_file.bin

    md5sum recover_binary_file.bin

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .