Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Today I had to remove the first 1131 bytes from an 800MB mixed text / binary file, a filtered subversion dump I'm hacking for a new repository. What's the best way to do this?

To start with I tried

dd bs=1 skip=1131 if=filtered.dump of=trimmed.dump

but after the skip this copies the remainder of the file a byte at a time, i.e. very slowly. In the end I worked out I needed 405 bytes to round this up to three blocks of 512 which I could skip

dd if=/dev/zero of=405zeros bs=1 count=405
cat 405zeros filtered.dump | dd bs=512 skip=3 of=trimmed.dump

which completed fairly quickly but there must have been a simpler / better way? Is there another tool I've forgotten about? Thanks!

share|improve this question
    
dd is the right tool for the job - looks like you came up with a nice, elegant solution to your problem. –  Justin Ethier Feb 3 '11 at 19:50

6 Answers 6

up vote 25 down vote accepted

You can switch bs and skip options:

dd bs=1131 skip=1 if=filtered.dump of=trimmed.dump

This way the operation can benefit from a greater block.

Otherwise, you could try with tail (although it's not safe to use it with binary files):

tail -c +1132 filtered.dump >trimmed.dump

Finally, you may use 3 dd instances to write something like this:

dd if=filtered.dump bs=512k | { dd bs=1131 count=1 of=/dev/null; dd bs=512k of=trimmed.dump; }

where the first dd prints its standard output filtered.dump; the second one just reads 1131 bytes and throws them away; then, the last one reads from its standard input the remaining bytes of filtered.dump and write them to trimmed.dump.

share|improve this answer
3  
Thanks! I didn't know that piped input carried over to a second process like that - that's very neat. I can't believe I didn't think of bs=1131 skip=1 though :-/ –  Rup Feb 4 '11 at 0:35
    
Most modern implementations of shell utilities work correctly with binary files (i.e. they have no trouble with null characters and won't insert an extra newline at the end of the file). Certainly GNU and *BSD implementations are safe. –  Gilles Feb 5 '11 at 0:04

You can use a sub-shell and two dd calls like this:

$ ( dd bs=1131 count=1 of=dev_null && dd bs=4K of=out.mp3 ) < 100827_MR029_LobbyControl.mp3
1+0 records in
1+0 records out
1131 bytes (1.1 kB) copied, 7.9691e-05 s, 14.2 MB/s
22433+1 records in
22433+1 records out
91886130 bytes (92 MB) copied, 0.329823 s, 279 MB/s
$ ls -l *
-rw------- 1 max users 91887261 2011-02-03 22:59 100827_MR029_LobbyControl.mp3
-rw-r--r-- 1 max users     1131 2011-02-03 23:04 dev_null
-rw-r--r-- 1 max users 91886130 2011-02-03 23:04 out.mp3
$ cat dev_null out.mp3 > orig
$ cmp 100827_MR029_LobbyControl.mp3 orig
share|improve this answer
    
Thanks - I didn't know piped input continued to a second process like that, I guess that's the sub shell? I'll definitely remember that! I've given Marco the tick because he got here first but +1 and thanks for the answer! –  Rup Feb 4 '11 at 0:36
    
@Rup, yes, the sub-shell - created via the parentheses - provides a stdin file descriptor and both dd calls successively consume the input from it. Yeah - Marco beat me by 29 seconds :) –  maxschlepzig Feb 4 '11 at 8:23

Yet another way to remove leading bytes from a file (without using dd at all) is to use xxd and sed or tail respectively.

bytes=$((1131*2))

xxd -p -c 256 filtered.dump | tr -d '\n' | sed "s/^.\{0,${bytes}\}//" | xxd -r -p > trimmed.dump

bytes=$((bytes + 1)) 
xxd -p -c 256 filtered.dump | tr -d '\n' | tail -c +${bytes} | xxd -r -p > trimmed.dump
share|improve this answer
    
That's neat, but I think I prefer just working with the file in binary rather than converting it to and from hex. –  Rup Apr 15 '13 at 14:56

Not sure when skip_bytes was added, but you have:

# echo {123456789}-abcdefgh- | 
                              dd bs=4096 skip=11 iflag=skip_bytes
-abcdefgh-
0+1 records in
0+1 records out
11 bytes (11 B) copied, 6.963e-05 s, 158 kB/s
share|improve this answer

This is the kind of situation where it pays to know a scripting language such as Python. Instead of wasting time fiddling around with the shell to do this simple task, you would just open the file in binary mode, skip the bytes, and copy. The scripting language library is sensible enough to notice that the last block is not full.

share|improve this answer
3  
To prove your point about wasting time with the shell and being more time efficient with a scripting language you could post the code (e.g. an one liner) in your favorite scripting language. –  maxschlepzig Feb 4 '11 at 8:26
3  
But now I know how to use dd properly I can't see how python could possibly be better. I have programmed this before - when I'm on Windows and don't have a port of dd to hand I usually bash out a quick program in C - but I can't see the benefit now. –  Rup Feb 4 '11 at 10:18
    
C is not a scripting language so I don't recommend this. Consider that you might stick this dd command in a shell script for future use. But in future the number of bytes to skip may be different in which case you are back to square one. But with a Python, PERL or Ruby script, you simply edit the skip count, and run it, or make it more general and have the skip count as an argument. In 197s it was necessary to figure out how to do the job without resorting to compiling C code, but today we have more powerful tools standard on UNIX OSes. –  Michael Dillon Feb 4 '11 at 22:22

@maxschlepzig asks for a online liner. Here is one in perl. It takes 2 argument: From byte and length. The input file must be given by '<' and the output will be on stdout:

perl -e 'sysseek(STDIN,shift,0) || die; $left = shift;
     while($read = sysread(STDIN,$buf, ($left > 32768 ? 32768 : $left))){
        $left -= $read; syswrite(STDOUT,$buf);
     }' 12345678901 19876543212 < bigfile > outfile

If length is larger than the file, the rest of the file will be copied.

On my system this delivers 3.5 GB/s.

share|improve this answer
    
I think his one-line challenge was to get you to prove that the scripting language solution was better than his one-line shell solution though. And I prefer his: it's shorter and clearer to me. If yours performs better it's because you're using a larger block size than him which is easily upped in his version too. –  Rup Apr 1 at 10:06
    
@Rup Alas, but no. You seem to forget that dd does not guarantee a full read. Try: yes | dd bs=1024k count=10 | wc unix.stackexchange.com/questions/17295/… –  Ole Tange Apr 1 at 14:56
    
Also my solution will not read the bytes you do not need (which could be several terabytes long). –  Ole Tange Apr 1 at 14:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.