9

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly 25% of the input file. How can I split up the file at the record boundary?

A naive approach would be zcat file | wc -l to get the line count, divide that number by 4 and then use split -l <number> file. However, this goes over the file twice and the line-counte is extremely slow (36mins). Is there a better way?

This comes close but is not what I am looking for. The accepted answer also does a line count.

EDIT:

The file contains sequencing data in fastq format. Two records look like this (anonymized):

@NxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGCGA+ATAGAGAG
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxTTTATGTTTTTAATTAATTCTGTTTCCTCAGATTGATGATGAAGTTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
AAAAA#FFFFFFFFFFFFAFFFFF#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF<AFFFFFFFFFFAFFFFFFFFFFFFFFFFFFF<FFFFFFFFFAFFFAFFAFFAFFFFFFFFAFFFFFFAAFFF<FAFAFFFFA
@NxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxGCGA+ATAGAGAG
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCCCTCTGCTGGAACTGACACGCAGACATTCAGCGGCTCCGCCGCCxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
AAAAA#FFFFF7FFFFFFAFFFFA#F7FFFFFFFFF7FFFFFAF<FFFFFFFFFFFFFFAFFF.F.FFFFF.FAFFF.FFFFFFFFFFFFFF.)F.FFA))FFF7)F7F<.FFFF.FFF7FF<.FFA<7FA.<.7FF.FFFAFF

Each record's first line starts with a @.

EDIT2:

zcat file > /dev/null takes 31mins.

EDIT3: Onlye the first line starts with @. None of the others will ever. See here. Records need to stay in order. It's not ok to add anything to the resulting file.

  • How long does a single zcat file > /dev/null take? – choroba Jun 16 '15 at 7:58
  • Can you provide a small sample of the file in question? – FloHimself Jun 16 '15 at 8:10
  • You say every record starts with @ and also that there are 4 lines per record. Are both of these absolute? - and can lines 2,3,4 start with @? and are there any non-record header of footer lines in the file? – Peter.O Jun 16 '15 at 10:52
  • 1
    Are you looking for a solution which handles compressed input and/or produces compressed output? Are you looking for four equally-sized compressed files? – Stephen Kitt Jun 16 '15 at 21:13
4

I don't think you can do this - not reliably, and not the way you ask. The thing is, the archive's compression ratio will probably not be evenly distributed from head to tail - the compression algorithm will apply better to some parts than others. That's just how it works. And so you can't factor your split on the size of the compressed file.

What's more, gzip just doesn't support storing the original size of compressed files greater than 4gbs in size - it can't handle it. And so you can't query the archive to get a reliable size - because it will fool you.

The 4 line thing - that's pretty easy, really. The 4-file thing - I just don't know how you could do it reliably and with an even distribution without first extracting the archive to get its uncompressed size. I don't think you can because I tried.

However, what you can do, is set a maximum size for split output files, and make sure those are always broken at record barriers. That you can easily do. Here's a little script that will do it by extracting the gzip archive, and piping the contents through a few explicit dd pipe-buffers with specific count=$rpt arguments, before passing that through lz4 to decompress/recompress each file on the fly. I also threw in a few little tee pipe tricks to print the last four lines for each segment to stderr as well.

(       IFS= n= c=$(((m=(k=1024)*k)/354))
        b=bs=354xk bs=bs=64k
        pigz -d </tmp/gz | dd i$bs o$b |
        while   read -r line _$((n+=1))
        do      printf \\n/tmp/lz4.$n\\n
        { {     printf %s\\n "$line"
                dd count=$c i$b o$bs
        }|      tee /dev/fd/3|lz4 -BD -9 >/tmp/lz4.$n
        } 3>&1| tail -n4 |tee /dev/fd/2 |
                wc -c;ls -lh /tmp/[gl]z*
        done
)

That will just keep going until it has handled all input. It doesn't attempt to split it by some percentage - which it cannot get - but instead it splits it per a maximum raw byte count per split. And anyway, a big part of your problem is that you can't get a reliable size on your archive because it is too big - whatever you do, don't do that again - make the splits less than 4gbs a piece this go round, maybe. This little script, at least, enables you to do this without having ever to write an uncompressed byte to disk.

Here's a shorter version stripped to the essentials - it doesn't add in all of the report stuff:

(       IFS= n= c=$((1024*1024/354))
        pigz -d | dd ibs=64k obs=354xk |
        while   read -r line _$((n+=1))
        do {    printf %s\\n "$line"
                dd count=$c obs=64k ibs=354xk
        }  |    lz4 -BD -9  >/tmp/lz4.$n
        done
)  </tmp/gz

It does all of the same things as the first, mostly, it just doesn't have as much to say about it. Also, there's less clutter so it's easier to see what's going on, maybe.

The IFS= thing is just to handle the one read line per iteration. We read one because we need our loop to end when input ends. This depends on your record-size - which, per your example, is 354 bytes per. I created a 4+gb gzip archive with some random data in order to test it.

The random data was got this way:

(       mkfifo /tmp/q; q="$(echo '[1+dPd126!<c]sc33lcx'|dc)"
        (tr '\0-\33\177-\377' "$q$q"|fold -b144 >/tmp/q)&
        tr '\0-\377' '[A*60][C*60][G*60][N*16][T*]' | fold -b144 |
        sed 'h;s/^\(.\{50\}\)\(.\{8\}\)/@N\1+\2\n/;P;s/.*/+/;H;x'|
        paste "-d\n" - - - /tmp/q| dd bs=4k count=kx2k  | gzip
)       </dev/urandom >/tmp/gz 2>/dev/null

...but maybe you don't need to worry so much about that, since you already have the data and all. Back to the solution...

Basically pigz - which seems to decompress a little faster than does zcat - pipes out the uncompressed stream, and dd buffers that output into write blocks sized specifically at a multiple of 354-bytes. The loop will read a $line once each iteration to test that input is still arriving, which it will printf afterward printf at lz4 before another dd is called to read blocks sized specifically at a multiple of 354-bytes - to synchronize with the buffering dd process - for the duration. There will be one short read per iteration because of the initial read $line - but that doesn't matter, because we're printing that at lz4 - our collector process - anyway.

I've set it up so each iteration will read approximately 1gb of uncompressed data and compress that in-stream to around 650Mb or so. lz4 is far faster than pretty much any other useful compression method - which is the reason I chose it here because I don't like to wait. xz would do a much better job at the actual compressing, probably, though. One thing about lz4, though, is it can often decompress at close to RAM speeds - which means a lot of times you can decompress an lz4 archive just fast as you would be able to write it into memory anyway.

The big one does a few reports per iteration. Both loops will print dd's report on the number of raw bytes transferred and the speed and so on. The big loop will also print the last 4 lines of input per cycle, and a byte count for same, followed by an ls of the directory to which I write the lz4 archives. Here are a couple of rounds of output:

/tmp/lz4.1
2961+1 records in
16383+1 records out
1073713090 bytes (1.1 GB) copied, 169.838 s, 6.3 MB/s
@NTACGTANTTCATTGGNATGACGCGCGTTTATGNGAGGGCGTCCGGAANGC+TCTCTNCC
TACGTANTTCATTGGNATGACGCGCGTTTATGNGAGGGCGTCCGGAANGCTCTCTNCCGAGCTCAGTATGTTNNAAGTCCTGANGNGTNGCGCCTACCCGACCACAACCTCTACTCGGTTCCGCATGCATGCAACACATCGTCA
+
I`AgZgW*,`Gw=KKOU:W5dE1m=-"9W@[AG8;<P7P6,qxE!7P4##,Q@c7<nLmK_u+IL4Kz.Rl*+w^A5xHK?m_JBBhqaLK_,o;p,;QeEjb|">Spg`MO6M'wod?z9m.yLgj4kvR~+0:.X#(Bf
354

-rw-r--r-- 1 mikeserv mikeserv 4.7G Jun 16 08:58 /tmp/gz
-rw-r--r-- 1 mikeserv mikeserv 652M Jun 16 12:32 /tmp/lz4.1

/tmp/lz4.2
2961+1 records in
16383+1 records out
1073713090 bytes (1.1 GB) copied, 169.38 s, 6.3 MB/s
@NTTGTTGCCCTAACCANTCCTTGGGAACGCAATGGTGTGANCTGCCGGGAC+CTTTTGCT
TTGTTGCCCTAACCANTCCTTGGGAACGCAATGGTGTGANCTGCCGGGACCTTTTGCTGCCCTGGTACTTTTGTCTGACTGGGGGTGCCACTTGCAGNAGTAAAAGCNAGCTGGTTCAACNAATAAGGACNANTTNCACTGAAC
+
>G-{N~Q5Z5QwV??I^~?rT+S0$7Pw2y9MV^BBTBK%HK87(fz)HU/0^%JGk<<1--7+r3e%X6{c#w@aA6Q^DrdVI0^8+m92vc>RKgnUnMDcU:j!x6u^g<Go?p(HKG@$4"T8BWZ<z.Xi
354

-rw-r--r-- 1 mikeserv mikeserv 4.7G Jun 16 08:58 /tmp/gz
-rw-r--r-- 1 mikeserv mikeserv 652M Jun 16 12:32 /tmp/lz4.1
-rw-r--r-- 1 mikeserv mikeserv 652M Jun 16 12:35 /tmp/lz4.2
  • gzip -l only works for <2GiB uncompressed files IIRC (something smaller than the OP's file anyway). – Stéphane Chazelas Jun 16 '15 at 14:25
  • @StéphaneChazelas - damn. That's the only way I could figure on getting an uncompressed size. Without that, this doesn't work at all. – mikeserv Jun 16 '15 at 14:27
4

Splitting files on the record boundaries is actually very easy, without any code:

zcat your_file.gz | split -l 10000 - output_name_

This will create output files of 10000 lines each, with names output_name_aa, output_name_ab, output_name_ac, ... With an input as large as yours, this will give you a lot of output files. Replace 10000 with any multiple of four, and you can make the output files as big or small as you like. Unfortunately, as with the other answers, there's not a good way to guarantee you'll get the desired number of (approximately) equal size of output files without making some guesses about the input. (Or actually piping the whole thing through wc.) If your records are approximately equal size (or at least, roughly evenly distributed) you can try coming up with an estimate like this:

zcat your_file.gz | head -n4000 | gzip | wc -c

That will tell you the compressed size of the first 1000 records of your file. Based on that, you can probably come up with an estimate of how many rows you want in each file to end up with four files. (If you don't want a degenerate fifth file left over, be sure to pad your estimate up a little bit, or be prepared to tack the fifth file onto the tail of the fourth.)

Edit: Here's one more trick, assuming you want compressed output files:

#!/bin/sh

base=$(basename $1 .gz)
unpigz -c $1 | split -l 100000 --filter='pigz -c > _$FILE.gz' - ${base}_

batch=$((`ls _*.gz | wc -l` / 4 + 1))
for i in `seq 1 4`; do
  files=`ls _*.gz | head -$batch`
  cat $files > ${base}_$i.gz && rm $files
done

This will create a lot of smaller files and then quickly cat them back together. (You may have to tweak the -l parameter depending on how long the lines in your files are.) It assumes you have a relatively recent version of GNU coreutils (for split --filter) and about 130% of your input file size in free disk space. Substitute gzip / zcat for pigz / unpigz if you don't have them. I've heard that some software libraries (Java?) can't handle gzip files concatenated this way, but I haven't had any problems with it so far. (pigz uses the same trick to parallelize compression.)

  • If you have pigz installed, you can speed things up a tiny bit by substituting 'pigz -cd' for 'zcat'. – Drew Jun 16 '15 at 23:07
  • 2
    Ah, I just noticed now that you already mentioned split in the question. But really, just about any solution is going to be doing about the same thing as split under the hood. The hard part is figuring out how many rows you need to put in each file. – Drew Jun 16 '15 at 23:15
3

From what I gather after checking the google-sphere, and further testing a 7.8 GiB .gz file, it seems that the metadata of the original uncompressed file's size is not accurate (ie. wrong) for large .gz files (greater than 4GiB (maybe 2GiB for some versions of gzip).
Re. my test of gzip's metadata:

* The compressed.gz file is  7.8 GiB ( 8353115038 bytes) 
* The uncompressed  file is 18.1 GiB (19436487168 bytes)
* The metadata says file is  2.1 GiB ( 2256623616 bytes) uncompressed

So it seems that it isn't possible to determe the uncompressed size without actually uncompressing it (which is a bit rough, to say the least!)

Anyhow, here is a way to split an uncompressed file at record boundaries, where each record contains 4 lines.

It uses the file's size in bytes (via stat), and with awk counting bytes (not characters). Whether or not line ending are LF | CR | CRLF, this script handles line ending length via builtin variable RT).

LC_ALL=C gawk 'BEGIN{"stat -c %s "ARGV[1] | getline inSize
                      segSiz=int(inSize/4)+((inSize%4)==0?0:1)
                      ouSplit=segSiz; segNb=0 }
               { lnb++; bytCt+=(length+length(RT))
                 print $0 > ARGV[1]"."segNb
                 if( lnb!=4 ) next
                 lnb=0
                 if( bytCt>=ouSplit ){ segNb++; ouSplit+=segSiz }
               }' myfile

Below is the test I used to check that each file's line count is mod 4 == 0

for i in myfile  myfile.{0..3}; do
    lc=$(<"$i" wc -l)
    printf '%s\t%s\t' "$i" $lc; 
    (( $(echo $lc"%4" | bc) )) && echo "Error: mod 4 remainder !" || echo 'mod 4 ok'  
done | column -ts$'\t' ;echo

Test output:

myfile    1827904  mod 4 ok
myfile.0  456976   mod 4 ok
myfile.1  456976   mod 4 ok
myfile.2  456976   mod 4 ok
myfile.3  456976   mod 4 ok

myfile was generated by:

printf %s\\n {A..Z}{A..Z}{A..Z}{A..Z}—{1..4} > myfile
2

This is not meant to be serious answer! I've been just toying with flex and this will most probably not work on a input file with ~50Gb (if at all, on larger input data than my test file):

This works for me on a ~1Gb file input.txt:

Given the flex input file splitter.l:

%{
#include <stdio.h>
extern FILE* yyin;
extern FILE* yyout;

int input_size = 0;

int part_num;
int part_num_max;
char **part_names;
%}

%%
@.+ {
        if (ftell(yyout) >= input_size / part_num_max) {
            fclose(yyout);
            if ((yyout = fopen(part_names[++part_num], "w")) == 0) {
                exit(1);
            }
        }
        fprintf(yyout, "%s", yytext);
    }
%%

int main(int argc, char *argv[]) {

    if (argc < 2) {
        return 1;
    } else if ((yyin = fopen(argv[1], "r")) == 0) {
        return 1;
    } else if ((yyout = fopen(argv[2], "w")) == 0) {
        fclose(yyin);
        return 1;
    } else {

        fseek(yyin, 0L, SEEK_END);
        input_size = ftell(yyin);
        rewind(yyin);

        part_num = 0;
        part_num_max = argc - 2;
        part_names = argv + 2;

        yylex();

        fclose(yyin);
        fclose(yyout);
        return 0;
    }
}

generating lex.yy.c and compiling it to the splitter binary with:

$ flex splitter.l && gcc lex.yy.c -ll -o splitter

Usage:

$ ./splitter input.txt output.part1 output.part2 output.part3 output.part4

Running time for 1Gb input.txt:

$ time ./splitter input.txt output.part1 output.part2 output.part3 output.part4

real    2m43.640s
user    0m48.100s
sys     0m1.084s
  • The actual lexing here is so simple, you really don't benefit from lex. Just call getc(stream) and apply some simple logic. Also, do you know that the . (dot) regex character in (f)lex matches any character except newline, right? Whereas these records are multi-line. – Kaz Jun 23 '15 at 20:03
  • @Kaz While your statements are generally corrent, this actually works with the data provided in Q. – FloHimself Jun 24 '15 at 5:32
  • Only accidentally, because there is a default rule when nothing matches: consume a character and print it to the output! In other rwords, you could do your file switching simply with a rule that recognizes the @ character, and then let the default rule copy the data. Now you have your rule copying part of the data as one big token, and then the default rule getting the second line one character at a time. – Kaz Jun 24 '15 at 19:38
  • Thanks for clarifying. I wonder, how would you solve this task with txr. – FloHimself Jun 24 '15 at 19:45
  • I'm not sure that I would because the task is to do a very simple thing with a large amount of data, as fast as possible. – Kaz Jun 24 '15 at 19:56
1

Here's a solution in Python that makes one pass over the input file writing the output files as it goes along.

A feature about using wc -l is that you are assuming each of the records here are the same size. That may be true here, but the solution below works even when that is not the case. It is basically using wc -c or the number of bytes in the file. In Python, this is done via os.stat()

So here's how the program works. We first compute the ideal split points as byte offsets. Then you read lines of the input file writing to the appropriate output file. When you see that you've exceeded the optimal next split point and you are at a record boundary, close of the last output file and open the next.

The program is optimal in this sense, it reads the bytes of the input file once; Getting the file size doesn't require reading the file data. Storage needed is proportional to the size of a line. But Python or the system presumably have reasonable file buffers to speed up I/O.

I've added parameters for how many files to split and what the record size is in case you want to adjust this in the future.

And clearly this could be translated to other programming languages as well.

One other thing, I'm not sure if Windows with its crlf handles the length of the line properly as it does on Unix-y systems. If len() is off by one here, I hope it's obvious how to adjust the program.

#!/usr/bin/env python
import os

# Adjust these
filename = 'file.txt'
rec_size = 4
file_splits = 4

size = os.stat(filename).st_size
splits = [(i+1)*size/file_splits for i in range(file_splits)]
with open(filename, 'r') as fd:
    linecount = 0
    i = 0 # File split number
    out = open('file%d.txt' % i, 'w')
    offset = 0  # byte offset of where we are in the file: 0..size
    r = 0 # where we are in the record: 0..rec_size-1
    for line in fd:
        linecount += 1
        r = (r+1) % rec_size
        if offset + len(line) > splits[i] and r == 1 :
            out.close()
            i += 1
            out = open('file%d.txt' % i, 'w')
        out.write(line)
        offset += len(line)
    out.close()
    print("file %s has %d lines" % (filename, linecount))
  • It's not splitting at a record boundary. eg. The first sub file split happens after the 3rd line with this input printf %s\\n {A..Z}{A..Z}{A..Z}{A..Z}—{1..4} – Peter.O Jun 16 '15 at 11:51
1

User FloHimself seemed curious about a TXR solution. Here is one using the embedded TXR Lisp:

(defvar splits 4)
(defvar name "data")

(let* ((fi (open-file name "r"))                 ;; input stream
       (rc (tuples 4 (get-lines fi)))            ;; lazy list of 4-tuples
       (sz (/ (prop (stat name) :size) splits))  ;; split size
       (i 1)                                     ;; split enumerator
       (n 0)                                     ;; tuplecounter within split
       (no `@name.@i`)                           ;; output split file name
       (fo (open-file no "w")))                  ;; output stream
  (whilet ((r (pop rc)))  ;; pop each 4-tuple
    (put-lines r fo) ;; send 4-tuple into output file
    ;; if not on the last split, every 1000 tuples, check the output file
    ;; size with stat and switch to next split if necessary.
    (when (and (< i splits)
               (> (inc n) 1000)
               (>= (seek-stream fo 0 :from-current) sz))
      (close-stream fo)
      (set fo (open-file (set no `@name.@(inc i)`) "w")
           n 0)))
  (close-stream fo))

Notes:

  1. For the same reason pop-ping each tuple from the lazy list of tuples is important, so that the lazy list is consumed. We must not retain a reference to the start of that list because then memory will grow as we march through the file.

  2. (seek-stream fo 0 :from-current) is no-op case of seek-stream, which makes itself useful by returning the current position.

  3. Performance: don't mention it. Usable, but won't be bringing any trophies home.

  4. Since we only do the size check every 1000 tuples, we could just make the tuple size 4000 lines.

0

If you don't need the new files to be contiguous chunks of the original file, you can do this entirely with sed in the following way:

sed -n -e '1~16,+3w1.txt' -e '5~16,+3w2.txt' -e '9~16,+3w3.txt' -e '13~16,+3w4.txt'

The -n stops it from printing each line, and each of the -e scripts is essentially doing the same thing. 1~16 matches the first line, and every 16th line after. ,+3 means match the next three lines after each of those. w1.txt says write all of those lines to the file 1.txt. This is taking every 4th group of 4 lines and writing it to a file, starting with the first group of 4 lines. The other three commands do the same thing, but they're each shifted forwards by 4 lines, and write to a different file.

This will break horribly if the file doesn't exactly match the specification you laid out, but otherwise it should work as you intended. I have not profiled it, so I don't know how efficient it will be, but sed is reasonably efficient at stream editing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.