61

Today I had to remove the first 1131 bytes from an 800MB mixed text / binary file, a filtered subversion dump I'm hacking for a new repository. What's the best way to do this?

To start with I tried

dd bs=1 skip=1131 if=filtered.dump of=trimmed.dump

but after the skip this copies the remainder of the file a byte at a time, i.e. very slowly. In the end I worked out I needed 405 bytes to round this up to three blocks of 512 which I could skip

dd if=/dev/zero of=405zeros bs=1 count=405
cat 405zeros filtered.dump | dd bs=512 skip=3 of=trimmed.dump

which completed fairly quickly but there must have been a simpler / better way? Is there another tool I've forgotten about? Thanks!

  • dd is the right tool for the job - looks like you came up with a nice, elegant solution to your problem. – Justin Ethier Feb 3 '11 at 19:50
62

You can switch bs and skip options:

dd bs=1131 skip=1 if=filtered.dump of=trimmed.dump

This way the operation can benefit from a greater block.

Otherwise, you could try with tail (although it's not safe to use it with binary files):

tail -c +1132 filtered.dump >trimmed.dump

Finally, you may use 3 dd instances to write something like this:

dd if=filtered.dump bs=512k | { dd bs=1131 count=1 of=/dev/null; dd bs=512k of=trimmed.dump; }

where the first dd prints its standard output filtered.dump; the second one just reads 1131 bytes and throws them away; then, the last one reads from its standard input the remaining bytes of filtered.dump and write them to trimmed.dump.

  • 6
    Thanks! I didn't know that piped input carried over to a second process like that - that's very neat. I can't believe I didn't think of bs=1131 skip=1 though :-/ – Rup Feb 4 '11 at 0:35
  • 2
    Most modern implementations of shell utilities work correctly with binary files (i.e. they have no trouble with null characters and won't insert an extra newline at the end of the file). Certainly GNU and *BSD implementations are safe. – Gilles Feb 5 '11 at 0:04
17

Not sure when skip_bytes was added, but to skip the first 11 bytes you have:

# echo {123456789}-abcdefgh- | 
                              dd bs=4096 skip=11 iflag=skip_bytes
-abcdefgh-
0+1 records in
0+1 records out
11 bytes (11 B) copied, 6.963e-05 s, 158 kB/s

Where iflag=skip_bytes tells dd to interpret the value for the skip option as bytes instead of blocks, making it straightforward.

  • Certainly a speed advantage for large files and a small amount of data to be removed. – sstn Nov 29 '17 at 11:08
  • This is the best answer, since it works for every block size e.g. iflag=skip_bytes skip=1234 bs=1M – phiresky Jan 15 '18 at 11:36
15

You can use a sub-shell and two dd calls like this:

$ ( dd bs=1131 count=1 of=dev_null && dd bs=4K of=out.mp3 ) < 100827_MR029_LobbyControl.mp3
1+0 records in
1+0 records out
1131 bytes (1.1 kB) copied, 7.9691e-05 s, 14.2 MB/s
22433+1 records in
22433+1 records out
91886130 bytes (92 MB) copied, 0.329823 s, 279 MB/s
$ ls -l *
-rw------- 1 max users 91887261 2011-02-03 22:59 100827_MR029_LobbyControl.mp3
-rw-r--r-- 1 max users     1131 2011-02-03 23:04 dev_null
-rw-r--r-- 1 max users 91886130 2011-02-03 23:04 out.mp3
$ cat dev_null out.mp3 > orig
$ cmp 100827_MR029_LobbyControl.mp3 orig
  • 1
    Thanks - I didn't know piped input continued to a second process like that, I guess that's the sub shell? I'll definitely remember that! I've given Marco the tick because he got here first but +1 and thanks for the answer! – Rup Feb 4 '11 at 0:36
  • 1
    @Rup, yes, the sub-shell - created via the parentheses - provides a stdin file descriptor and both dd calls successively consume the input from it. Yeah - Marco beat me by 29 seconds :) – maxschlepzig Feb 4 '11 at 8:23
6

If the filesystem and the Linux kernel support it then you could try fallocate if you want to make the changes inplace: in the best case there is no data IO at all:

$ fallocate <magic> -o 0 -l 1131 inplace.dump

where <magic> depends on the filesystem, Linux version, and the file type (FALLOC_FL_COLLAPSE_RANGE or FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE could be used internally).

3

You should use count=0 - that's a simple lseek() whenever possible.

Like this:

{  dd bs=1131 skip=1 count=0; cat; } <filtered.dump >trimmed.dump

dd will lseek() the input file descriptor to an 1131 byte offset, and then cat will simply copy whatever remains to output.

2

Yet another way to remove leading bytes from a file (without using dd at all) is to use xxd and sed or tail respectively.

bytes=$((1131*2))

xxd -p -c 256 filtered.dump | tr -d '\n' | sed "s/^.\{0,${bytes}\}//" | xxd -r -p > trimmed.dump

bytes=$((bytes + 1)) 
xxd -p -c 256 filtered.dump | tr -d '\n' | tail -c +${bytes} | xxd -r -p > trimmed.dump
  • That's neat, but I think I prefer just working with the file in binary rather than converting it to and from hex. – Rup Apr 15 '13 at 14:56
2

@maxschlepzig asks for a online liner. Here is one in perl. It takes 2 argument: From byte and length. The input file must be given by '<' and the output will be on stdout:

perl -e 'sysseek(STDIN,shift,0) || die; $left = shift;
     while($read = sysread(STDIN,$buf, ($left > 32768 ? 32768 : $left))){
        $left -= $read; syswrite(STDOUT,$buf);
     }' 12345678901 19876543212 < bigfile > outfile

If length is larger than the file, the rest of the file will be copied.

On my system this delivers 3.5 GB/s.

  • I think his one-line challenge was to get you to prove that the scripting language solution was better than his one-line shell solution though. And I prefer his: it's shorter and clearer to me. If yours performs better it's because you're using a larger block size than him which is easily upped in his version too. – Rup Apr 1 '14 at 10:06
  • @Rup Alas, but no. You seem to forget that dd does not guarantee a full read. Try: yes | dd bs=1024k count=10 | wc unix.stackexchange.com/questions/17295/… – Ole Tange Apr 1 '14 at 14:56
  • Also my solution will not read the bytes you do not need (which could be several terabytes long). – Ole Tange Apr 1 '14 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.