Split large file into chunks without splitting entry

Question

I have a rather large .msg file formatted in the UIEE format.

$ wc -l big_db.msg
8726593 big_db.msg

Essentially, the file is made up of entries of various length that look something like this:

UR|1
AA|Condon, Richard
TI|Prizzi's Family
CN|Collectable- Good/Good
MT|FICTION
PU|G.P. Putnam & Sons
DP|1986
ED|First Printing.
BD|Hard Cover
NT|0399132104
KE|MAFIA
KE|FICTION
PR|44.9
XA|4
XB|1
XC|BO
XD|S

UR|10
AA|Gariepy, Henry
TI|Portraits of Perseverance
CN|Good/No Jacket
MT|SOLD
PU|Victor Books
DP|1989
BD|Mass Market Paperback
NT|1989 tpb g 100 meditations from the Book of Job "This book...help you
NT| persevere through the struggles of your life..."
KE|Bible
KE|religion
KE|Job
KE|meditations
PR|28.4
XA|4
XB|5
XC|BO
XD|S

This is an examples of two entries, separated by a blank line. I wish to split this big file into smaller files without breaking an entry into two files.

Each individual entry is separated by a newline (a completely blank line) in the file. I wish to break this 8.7 million line file into 15 files. I understand that tools like split exist but I'm not quite sure how to split the file but only have it split on a newline so a single entry doesn't get broken into multiple files.

Answer 1

Using the suggestion of csplit:

Splitting based on line numbers

$ csplit file.txt <num lines> "{repetitions}"

Example

Say I have a file with 1000 lines in it.

$ seq 1000 > file.txt

$ csplit file.txt 100 "{8}"
288
400
400
400
400
400
400
400
400
405

results in files like so:

$ wc -l xx*
  99 xx00
 100 xx01
 100 xx02
 100 xx03
 100 xx04
 100 xx05
 100 xx06
 100 xx07
 100 xx08
 101 xx09
   1 xx10
1001 total

You can get around the static limitation of having to specify the number of repetitions by pre-calculating the numbers based on the number of lines in your particular file ahead of time.

$ lines=100
$ echo $lines 
100

$ rep=$(( ($(wc -l file.txt | cut -d" " -f1) / $lines) -2 ))
$ echo $rep
8

$ csplit file.txt 100 "{$rep}"
288
400
400
400
400
400
400
400
400
405

Splitting based on blank lines

If on the other hand you'd like to simply split up a file on blank lines that are contained in the file you can use this version of split:

$ csplit file2.txt '/^$/' "{*}"

Example

Say I've added 4 blank lines to the file.txt above, and make the file file2.txt. You can see that they've been manually added like so:

$ grep -A1 -B1 "^$" file2.txt
20

21
--
72

73
--
112

113
--
178

179

The above shows that I've added them between the corresponding numbers within my sample file. Now when I run the csplit command:

$ csplit file2.txt '/^$/' "{*}"
51
157
134
265
3290

You can see that I now have 4 files that have been split up based on the blank line:

$ grep -A1 -B1 '^$' xx0*
xx01:
xx01-21
--
xx02:
xx02-73
--
xx03:
xx03-113
--
xx04:
xx04-179

References

• Unix Powertools - 35.10 Splitting Files by Context: csplit
• csplit : Split a file based on patterns
• How to split file on first empty line in a portable way in shell (e.g. using sed)?

Answer 2

If you don't care about the orders of the records, you could do:

gawk -vRS= '{printf "%s", $0 RT > "file.out." (NR-1)%15}' file.in

Otherwise, you'd need first to get the number of records first, to know how many to put in each output file:

gawk -vRS= -v "n=$(gawk -vRS= 'END {print NR}' file.in)" '
  {printf "%s", $0 RT > "file.out." int((NR-1)*15/n)}' file.in

Answer 3

Here's a solution that could work:

seq 1 $(((lines=$(wc -l </tmp/file))/16+1)) $lines |
sed 'N;s|\(.*\)\(\n\)\(.*\)|\1d;\1,\3w /tmp/uptoline\3\2\3|;P;$d;D' |
sed -ne :nl -ne '/\n$/!{N;bnl}' -nf - /tmp/file

It works by allowing the first sed to write the second sed's script. The second sed first gathers all input lines until it encounters a blank line. It then writes all output lines to a file. The first sed writes out a script for the second one instructing it on where to write its output. In my test case that script looked like this:

1d;1,377w /tmp/uptoline377
377d;377,753w /tmp/uptoline753
753d;753,1129w /tmp/uptoline1129
1129d;1129,1505w /tmp/uptoline1505
1505d;1505,1881w /tmp/uptoline1881
1881d;1881,2257w /tmp/uptoline2257
2257d;2257,2633w /tmp/uptoline2633
2633d;2633,3009w /tmp/uptoline3009
3009d;3009,3385w /tmp/uptoline3385
3385d;3385,3761w /tmp/uptoline3761
3761d;3761,4137w /tmp/uptoline4137
4137d;4137,4513w /tmp/uptoline4513
4513d;4513,4889w /tmp/uptoline4889
4889d;4889,5265w /tmp/uptoline5265
5265d;5265,5641w /tmp/uptoline5641

I tested it like this:

printf '%s\nand\nmore\nlines\nhere\n\n' $(seq 1000) >/tmp/file

This provided me a file of 6000 lines, which looked like this:

<iteration#>
and
more
lines
here
#blank

...repeated 1000 times.

After running the script above:

set -- /tmp/uptoline*
echo $# total splitfiles
for splitfile do
    echo $splitfile
    wc -l <$splitfile
    tail -n6 $splitfile
done    

OUTPUT

15 total splitfiles
/tmp/uptoline1129
378
188
and
more
lines
here

/tmp/uptoline1505
372
250
and
more
lines
here

/tmp/uptoline1881
378
313
and
more
lines
here

/tmp/uptoline2257
378
376
and
more
lines
here

/tmp/uptoline2633
372
438
and
more
lines
here

/tmp/uptoline3009
378
501
and
more
lines
here

/tmp/uptoline3385
378
564
and
more
lines
here

/tmp/uptoline3761
372
626
and
more
lines
here

/tmp/uptoline377
372
62
and
more
lines
here

/tmp/uptoline4137
378
689
and
more
lines
here

/tmp/uptoline4513
378
752
and
more
lines
here

/tmp/uptoline4889
372
814
and
more
lines
here

/tmp/uptoline5265
378
877
and
more
lines
here

/tmp/uptoline5641
378
940
and
more
lines
here

/tmp/uptoline753
378
125
and
more
lines
here

Answer 4

If you are looking to split only at the end of a line, you should be able to do it with the -l option for split.

If you are looking to split on a blank line (\n\n), here is how I would do it in ksh. I haven't tested it, and it probably isn't ideal, but something along this line would work:

filenum=0
counter=0
limit=580000

while read LINE
do
  counter=counter+1

  if (( counter >= limit ))
  then
    if [[ $LINE == "" ]]
    then
      filenum=filenum+1
      counter=0
    fi
  fi

  echo $LINE >>big_db$filenum.msg
done <big_db.msg

Answer 5

Try awk

awk 'BEGIN{RS="\n\n"}{print $0 > FILENAME"."FNR}' big_db.msg

Answer 6

If you don't care about the order of the records but you're particular about getting a certain number of output files, Stephane's answer is the way I'd go. But I have a feeling you might care more about specifying a size that each output file should not exceed. That actually makes it easier because you can read through your input file and collect records until you reach that size, and then start a new output file. If that works for you, most programming languages can handle your task with a short script. Here's an awk implementation:

BEGIN {
    RS = "\n\n"
    ORS = "\n\n"
    maxlen = (maxlen == 0 ? 500000 : maxlen)
    oi = 1
}

{
    reclen = length($0) + 2
    if (n + reclen > maxlen) {
        oi++
        n = 0
    }
    n += reclen
    print $0 > FILENAME"."oi
}

Put this in a file, say program.awk, and run it with awk -v maxlen=10000 -f program.awk big_db.msg where the value of maxlen is the most bytes you want in any one file. It will use 500k as a default.

If you want to get a set number of files, probably the easiest way is to just divide the size of your input file by the number of files you want, and then add a bit to that number to get maxlen. For example, to get 15 files out of your 8726593 bytes, divide by 15 to get 581773, and add some, so maybe give maxlen=590000 or maxlen=600000. If you want to do this repeatably, it would be possible to configure the program to do it.

Answer 7

split -n l/100 input_file prefix_name

This will split a large file named input_file into 100 files with name prefix_nameXX without cutting lines.

Answer 8

With GNU Parallel it would look something like this:

parallel --pipepart -a bigfile --recend '\n\n' --block -1 --jobs 15 'cat >{#}'

This will start 15 cats in parallel. If you only want one cat:

parallel --pipepart -a bigfile --recend '\n\n' --block -15 --jobs 1 'cat >{#}'

It may (or may not) make sense to replace cat with the upload program.