0

Let's say I have a large file (several gigs) with n lines in it. I would like to add/insert a line after k bytes offset, from the beginning of the file, then what would be the fastest way to achieve that?

  • what did you try until now? – yael Mar 10 at 5:53
  • please let us know why you want to do it? – yael Mar 10 at 5:54
  • I was thinking of trying something like head -c k file > temp.log; echo 'some data\n' >> temp.log; tail -c +(k+1) file > temp.log; mv temp.log file; but not sure if its the best way to do this simple task.. – khan Mar 10 at 5:56
  • 1
    @yael it doesn't really matter why he wants to do it. – RonJohn Mar 10 at 6:38
  • 1
    For Very Large Files, your standard Unix text tools might not be the best solution. It certainly can't hurt to try head -c k file > temp.log; echo 'some data\n' >> temp.log; tail -c +(k+1) file > temp.log; mv temp.log file especially if it's a one-time operation. It might have already completed by now. – RonJohn Mar 10 at 7:32
2

Here's a Python solution:

#!/usr/bin/env python3
# -*- encoding: utf-8 -*-
"""split_bytes.py"""

import os
import sys

stdout = os.fdopen(sys.stdout.fileno(), 'wb')

path_to_file = sys.argv[1]
width_in_bytes = int(sys.argv[2])

with open(path_to_file, "rb") as f:
    byte = f.read(1)
    while byte:
        for i in range(width_in_bytes):
            stdout.write(byte)
            byte = f.read(1)
        stdout.write(b"\n")

You could execute it like this:

python split_bytes.py path/to/file offset > new_file

As a test, I generated a 1GB file of random data:

dd if=/dev/urandom of=data.bin bs=64M count=16 iflag=fullblock

Then ran the script on that file:

python split_lines.py data.bin 10 > split-data.bin
  • This looks like a promising approach..let me try some tests. – khan Mar 11 at 3:05
0

a bash only solution :

use split command :

split --lines=2 --suffix-length=6 /etc/passwd /tmp/split.passwd.part

reassemble the file into one new

(
  for F in /tmp/split.passwd.part* ; 
  do 
    cat $F ; 
    echo ; 
  done
) > /tmp/passwd_emptyline_evrey_2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.