4

I need to delete the first 2 bytes of a 6MB file. However, this is an embedded Linux with only 32 Mbytes RAM and less than 1 MB free flash memory.

I tried using dd, as:

1 - # dd bs=1 skip=2 count=1022 if=input of=ouput_1

2 - # dd bs=1024 skip=1 if=input of=ouput_2

3 - # rm -rf input

4 - # (dd if=ouput_1 ; dd if=ouput_2) > ouput

5 - # rm -rf ouput_1 ouput_2

With all files under the /tmp (mounted as tmpfs on RAM), my problem is that just before lines 3 and 5, the memory needed is 12 Mbyte (2x6MB), and the process sometimes fail and gives an "Not enough memory" error.

Is there a way I can remove the first 2 bytes without allocating twice the size of the file ? Can I use dd (or any other) to cut a binary file 'in place' ?

  • You might need to do this at the Unix/C programming level. Sorry, I don't have specific ideas how do do this. – Faheem Mitha Feb 26 '15 at 23:39
  • 1
    See Best way to remove bytes from the start of a file? but beware that simple solutions leave the file in an indeterminate state if interrupted midway. If you need the operation to not hose the file on power failure, it gets more complicated. – Gilles Feb 26 '15 at 23:49
  • 1
    @DigitalTrauma Ah, true, the other question doesn't discuss the in-place aspect. unix.stackexchange.com/questions/11067/… on the other hand is a duplicate. – Gilles Feb 27 '15 at 0:10
  • 1
    @DigitalTrauma grep was just an example in that earlier question, the difficulty is the edit in place. However robustness to power failures is a twist that I think has never come up on this site. – Gilles Feb 27 '15 at 0:17
  • 1
    Somehow I have a feeling that "editing file in-place" and "resistant against sudden power failure" are two opposing requirements that can't be satisfied together. – Abel Cheung Feb 27 '15 at 1:39
1

I think this should work:

$   # Create test file
$ echo "Hello, World" > h.data
$
$   # Move contents up by 2 bytes
$   # Note if= and of= are the same for in-place editing
$ dd bs=2 if=h.data skip=1 seek=0 conv=notrunc of=h.data
5+1 records in
5+1 records out
11 bytes (11 B) copied, 0.000598796 s, 18.4 kB/s
$
$   # Note 11 bytes were moved above
$   # Truncate the file after byte 11
$ dd bs=11 if=h.data skip=1 seek=1 count=0 of=h.data
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000338852 s, 0.0 kB/s
$
$   # Display edited file:
$ cat h.data
llo, World
$ 

Wrapping this all up in a script you could have something like this:

#!/bin/bash

size=$(stat -c %s "$2")
dd bs=$1 if="$2" skip=1 seek=0 conv=notrunc of="$2"
dd bs=$((size - $1)) if="$2" skip=1 seek=1 count=0 of="$2"

Call this as:

./truncstart.sh 2 file.dat

Where 2 is the number of bytes to delete from the beginning of file.data


As @Gilles points out, this solution is not robust in case of unplanned outage, which could occur part-way through dd's processing; in which case the file would be corrupted.

  • Thanks for the answer, it is very helpful. Although the script removed the first two bytes, it did not quite solve the problem, as the last line's bs=$((size - $1)) also tries to allocate large chunk in the memory and dd failed with "Out of memory". I ended up using a small utility with ftruncate() to truncate the last 2 bytes. – srd Feb 27 '15 at 19:56
  • I think changing it to this should work, though will be slower: `dd bs=1 if="$2" skip=$((size - $1)) seek=$((size - $1)) count=0 of="$2" – Digital Trauma Feb 28 '15 at 2:28
  • Yeah, I tried that and it was VERY slow, 2 bytes at a time for 6 MB! That's why I used the little truncate program. – srd Mar 2 '15 at 22:47
0

Based on @DigitalTrauma answer, this is what worked for me at last:

size=$(stat -c %s file)
dd bs=2 if=file skip=1 seek=0 conv=notrunc count=511 of=file
dd if=file ibs=1024 skip=1 of=file conv=notrunc obs=1022 seek=1
truncate file $(( size - 2 ))

Removing the first two bytes is done over 2 dd steps to speed things up and truncate is a small utility to truncate the last few bytes from the file:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>


int main(int argc, char **argv) 
{
    if (argc != 3 ) {
        printf ("Usage: %s <file> <bytes>\n", argv[0]);
        exit(1);
    }

    if (truncate(argv[1], atoi(argv[2]))) {
        printf (" Error ! \n");
        exit(1);
    }

    return(0);
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.