1

I want to take a huge file data.txt and:

  1. Split it by 256m
  2. Add the first 3 lines of data.txt to each split file
  3. gzip it

data.txt looks like:

aaa
bbb
ccc
<data>
<data>
<data>
...
<few million rows>
<data>

The end result would be that each split file will have the same first 3 lines

aaa
bbb
ccc

The most efficient way I know how to do it is in the memory. If I leave step 2 it would look like:

split --bytes=256M -d -a 3 --filter='gzip > $FILE.gz' data.txt split/data_

But with step 2 I'm not sure how to do it without split the gzip into a separate command, thus writing the files to disk.

1 Answer 1

4

You can make the filter more complex:

split -C 256M -d -a 3 --filter '(head -n 3 data.txt; cat) | gzip > $FILE.gz' <(tail -n +4 data.txt) data_

This skips the first three lines of data.txt, and then reintroduces them into every split section.

I’m using -C instead of --bytes to ensure that the split files contain full records.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .