8

I have a rather large .msg file formatted in the UIEE format.

$ wc -l big_db.msg
8726593 big_db.msg

Essentially, the file is made up of entries of various length that look something like this:

UR|1
AA|Condon, Richard
TI|Prizzi's Family
CN|Collectable- Good/Good
MT|FICTION
PU|G.P. Putnam & Sons
DP|1986
ED|First Printing.
BD|Hard Cover
NT|0399132104
KE|MAFIA
KE|FICTION
PR|44.9
XA|4
XB|1
XC|BO
XD|S

UR|10
AA|Gariepy, Henry
TI|Portraits of Perseverance
CN|Good/No Jacket
MT|SOLD
PU|Victor Books
DP|1989
BD|Mass Market Paperback
NT|1989 tpb g 100 meditations from the Book of Job "This book...help you
NT| persevere through the struggles of your life..."
KE|Bible
KE|religion
KE|Job
KE|meditations
PR|28.4
XA|4
XB|5
XC|BO
XD|S

This is an examples of two entries, separated by a blank line. I wish to split this big file into smaller files without breaking an entry into two files.

Each individual entry is separated by a newline (a completely blank line) in the file. I wish to break this 8.7 million line file into 15 files. I understand that tools like split exist but I'm not quite sure how to split the file but only have it split on a newline so a single entry doesn't get broken into multiple files.

10
  • csplit also exists.
    – mikeserv
    Jun 20, 2014 at 18:38
  • Can you create temporary files?
    – Braiam
    Jun 20, 2014 at 18:46
  • @Braiam, not sure what you mean but I think so. I have full access over the file system. Jun 20, 2014 at 18:47
  • he means creating files that are used temporary for the process
    – polym
    Jun 20, 2014 at 18:49
  • 1
    Why exactly 15 files, if I may ask? Are the prefixes before the pipe | (like UR, AA, TI) relevant for the count of files, even the same to be exact?
    – polym
    Jun 20, 2014 at 18:52

8 Answers 8

3

Using the suggestion of csplit:

Splitting based on line numbers

$ csplit file.txt <num lines> "{repetitions}"

Example

Say I have a file with 1000 lines in it.

$ seq 1000 > file.txt

$ csplit file.txt 100 "{8}"
288
400
400
400
400
400
400
400
400
405

results in files like so:

$ wc -l xx*
  99 xx00
 100 xx01
 100 xx02
 100 xx03
 100 xx04
 100 xx05
 100 xx06
 100 xx07
 100 xx08
 101 xx09
   1 xx10
1001 total

You can get around the static limitation of having to specify the number of repetitions by pre-calculating the numbers based on the number of lines in your particular file ahead of time.

$ lines=100
$ echo $lines 
100

$ rep=$(( ($(wc -l file.txt | cut -d" " -f1) / $lines) -2 ))
$ echo $rep
8

$ csplit file.txt 100 "{$rep}"
288
400
400
400
400
400
400
400
400
405

Splitting based on blank lines

If on the other hand you'd like to simply split up a file on blank lines that are contained in the file you can use this version of split:

$ csplit file2.txt '/^$/' "{*}"

Example

Say I've added 4 blank lines to the file.txt above, and make the file file2.txt. You can see that they've been manually added like so:

$ grep -A1 -B1 "^$" file2.txt
20

21
--
72

73
--
112

113
--
178

179

The above shows that I've added them between the corresponding numbers within my sample file. Now when I run the csplit command:

$ csplit file2.txt '/^$/' "{*}"
51
157
134
265
3290

You can see that I now have 4 files that have been split up based on the blank line:

$ grep -A1 -B1 '^$' xx0*
xx01:
xx01-21
--
xx02:
xx02-73
--
xx03:
xx03-113
--
xx04:
xx04-179

References

12
  • I edited the OP with my attempt to use this and I could not get it to work. Jun 20, 2014 at 19:27
  • The file wasn't split on a new, blank line, which is what I've been trying to accomplish. Jun 20, 2014 at 19:32
  • @user2036066 - you want to split the file in to 15 file chunks making sure there is not splitting on a partial line or something else?
    – slm
    Jun 20, 2014 at 19:32
  • @user2036066 - wait so the file has 14-15 completely blank lines throughout it that you want to split on?
    – slm
    Jun 20, 2014 at 19:33
  • Edited the op again with more context @slm Jun 20, 2014 at 19:35
3

If you don't care about the orders of the records, you could do:

gawk -vRS= '{printf "%s", $0 RT > "file.out." (NR-1)%15}' file.in

Otherwise, you'd need first to get the number of records first, to know how many to put in each output file:

gawk -vRS= -v "n=$(gawk -vRS= 'END {print NR}' file.in)" '
  {printf "%s", $0 RT > "file.out." int((NR-1)*15/n)}' file.in
2
  • Using awk to split on blank lines was my first thought, too - +1
    – godlygeek
    Jun 20, 2014 at 20:13
  • What are file.in and file.out?
    – mikeserv
    Jun 20, 2014 at 21:27
2

Here's a solution that could work:

seq 1 $(((lines=$(wc -l </tmp/file))/16+1)) $lines |
sed 'N;s|\(.*\)\(\n\)\(.*\)|\1d;\1,\3w /tmp/uptoline\3\2\3|;P;$d;D' |
sed -ne :nl -ne '/\n$/!{N;bnl}' -nf - /tmp/file

It works by allowing the first sed to write the second sed's script. The second sed first gathers all input lines until it encounters a blank line. It then writes all output lines to a file. The first sed writes out a script for the second one instructing it on where to write its output. In my test case that script looked like this:

1d;1,377w /tmp/uptoline377
377d;377,753w /tmp/uptoline753
753d;753,1129w /tmp/uptoline1129
1129d;1129,1505w /tmp/uptoline1505
1505d;1505,1881w /tmp/uptoline1881
1881d;1881,2257w /tmp/uptoline2257
2257d;2257,2633w /tmp/uptoline2633
2633d;2633,3009w /tmp/uptoline3009
3009d;3009,3385w /tmp/uptoline3385
3385d;3385,3761w /tmp/uptoline3761
3761d;3761,4137w /tmp/uptoline4137
4137d;4137,4513w /tmp/uptoline4513
4513d;4513,4889w /tmp/uptoline4889
4889d;4889,5265w /tmp/uptoline5265
5265d;5265,5641w /tmp/uptoline5641

I tested it like this:

printf '%s\nand\nmore\nlines\nhere\n\n' $(seq 1000) >/tmp/file

This provided me a file of 6000 lines, which looked like this:

<iteration#>
and
more
lines
here
#blank

...repeated 1000 times.

After running the script above:

set -- /tmp/uptoline*
echo $# total splitfiles
for splitfile do
    echo $splitfile
    wc -l <$splitfile
    tail -n6 $splitfile
done    

OUTPUT

15 total splitfiles
/tmp/uptoline1129
378
188
and
more
lines
here

/tmp/uptoline1505
372
250
and
more
lines
here

/tmp/uptoline1881
378
313
and
more
lines
here

/tmp/uptoline2257
378
376
and
more
lines
here

/tmp/uptoline2633
372
438
and
more
lines
here

/tmp/uptoline3009
378
501
and
more
lines
here

/tmp/uptoline3385
378
564
and
more
lines
here

/tmp/uptoline3761
372
626
and
more
lines
here

/tmp/uptoline377
372
62
and
more
lines
here

/tmp/uptoline4137
378
689
and
more
lines
here

/tmp/uptoline4513
378
752
and
more
lines
here

/tmp/uptoline4889
372
814
and
more
lines
here

/tmp/uptoline5265
378
877
and
more
lines
here

/tmp/uptoline5641
378
940
and
more
lines
here

/tmp/uptoline753
378
125
and
more
lines
here
1

If you are looking to split only at the end of a line, you should be able to do it with the -l option for split.

If you are looking to split on a blank line (\n\n), here is how I would do it in ksh. I haven't tested it, and it probably isn't ideal, but something along this line would work:

filenum=0
counter=0
limit=580000

while read LINE
do
  counter=counter+1

  if (( counter >= limit ))
  then
    if [[ $LINE == "" ]]
    then
      filenum=filenum+1
      counter=0
    fi
  fi

  echo $LINE >>big_db$filenum.msg
done <big_db.msg
5
  • 1
    It's possible I've misread, but op is asking how to split on \n\n, I think.
    – mikeserv
    Jun 20, 2014 at 18:40
  • That doesn't really help me because that will still split the file mid-entry. I need it so the file will only be split on a blank line. Jun 20, 2014 at 18:40
  • Yes I did misread, sorry. It may not be the best way, I would just read in the original file into a loop with a counter of how many lines you have passed, and once you hit the number you want to split on start outputting to a new file at the next blank line.
    – hornj
    Jun 20, 2014 at 18:42
  • Attempting to test this script right now. Jun 20, 2014 at 19:10
  • 1
    I think OP is not asking how to split on \n\n, but rather not to split in midst of a line. He is calling a newline a blank line.
    – polym
    Jun 20, 2014 at 19:11
0

Try awk

awk 'BEGIN{RS="\n\n"}{print $0 > FILENAME"."FNR}' big_db.msg
2
  • Attempting this solution right now Jun 20, 2014 at 19:10
  • 2
    This solution creates a new file for every entry, which is not what I want at all. Jun 20, 2014 at 19:11
0

If you don't care about the order of the records but you're particular about getting a certain number of output files, Stephane's answer is the way I'd go. But I have a feeling you might care more about specifying a size that each output file should not exceed. That actually makes it easier because you can read through your input file and collect records until you reach that size, and then start a new output file. If that works for you, most programming languages can handle your task with a short script. Here's an awk implementation:

BEGIN {
    RS = "\n\n"
    ORS = "\n\n"
    maxlen = (maxlen == 0 ? 500000 : maxlen)
    oi = 1
}

{
    reclen = length($0) + 2
    if (n + reclen > maxlen) {
        oi++
        n = 0
    }
    n += reclen
    print $0 > FILENAME"."oi
}

Put this in a file, say program.awk, and run it with awk -v maxlen=10000 -f program.awk big_db.msg where the value of maxlen is the most bytes you want in any one file. It will use 500k as a default.

If you want to get a set number of files, probably the easiest way is to just divide the size of your input file by the number of files you want, and then add a bit to that number to get maxlen. For example, to get 15 files out of your 8726593 bytes, divide by 15 to get 581773, and add some, so maybe give maxlen=590000 or maxlen=600000. If you want to do this repeatably, it would be possible to configure the program to do it.

0
split -n l/100 input_file prefix_name

This will split a large file named input_file into 100 files with name prefix_nameXX without cutting lines.

0

With GNU Parallel it would look something like this:

parallel --pipepart -a bigfile --recend '\n\n' --block -1 --jobs 15 'cat >{#}'

This will start 15 cats in parallel. If you only want one cat:

parallel --pipepart -a bigfile --recend '\n\n' --block -15 --jobs 1 'cat >{#}'

It may (or may not) make sense to replace cat with the upload program.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .