1

I have a 2Gb file. That has a a header and many-many "event"-structures. This is how the beginning will look:

<run example>
<header>
 5
   This is header                                                   
</header>

<event  =             22>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
       0       0       0       0       0       1       1       8
       0       1       0       2       1       5       2       0
       2       1       3       7       3       1       1       0
       1       0   10100       2       3       1       5       1
       1       5       1       7       2       3       2       2
</evhead>
 0 97
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0  4.63012694434E+00  2.62561831936E+00 -2.31855757639E+01  2.37887130977E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      22       1       0       0       0 -4.63012694434E+00 -2.62561831936E+00 -4.31442423592E+00  3.71128690719E+00 -5.75956188088E+00
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       2       0       0       0 -2.16995636615E-14 -1.11022302463E-15  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      22       4       0       0       0 -4.60626572550E+00 -2.61208727495E+00 -2.23619853289E+00  5.74815342040E+00  0.00000000000E+00
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
</event>

The whole file contains 97000 of such "event" blocks. So I want to split this file in 10 files in the way that each would contain the header AND 10 000 of the "event" blocks. All blocks differ by index (they are random). The last file will of course contain only 7 000 blocks.

I have tried multiple instructions from stack like here: https://stackoverflow.com/questions/8544197/splitting-a-file-in-linux-based-on-content https://stackoverflow.com/questions/8544197/splitting-a-file-in-linux-based-on-content but non did worked for me.

Below is a bit bigger example of the file for any tests (file_to_download):

<run example>
<header>
 5
   header                                                     
</header>

<event  =             22>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
</evhead>
 0 97
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
</event>

<event  =             26>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
</evhead>
 0 52
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
</event>

<event  =             31>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
       0       0       0       0       0       1       1       8
</evhead>
 0 92
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0  4.39003604933E+00  4.97037860337E+00 -2.04926313413E+01  2.15389187176E+01  5.10000000000E-04
</event>

<event  =             37>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
</evhead>
 0 77
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0  7.91768942174E+00  3.75815788575E+00 -2.09569980000E+01  2.27158385693E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
</event>

<event  =             41>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
</evhead>
 0 122
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0 -3.63469912393E+00  3.95372353695E+00 -1.62133507727E+01  1.70796870892E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
</event>
  • Is it an XML file? The snippet looks like one. Have you considering using an XML parser to split out this content? If not, is there any reason why the answer has to include awk? – roaima Oct 2 '16 at 18:43
  • 1
    @roaima actually any native linux utility will do. I am using a computer where I am not allowed to instal any software – greyxray Oct 2 '16 at 19:35
2

With GNU awk:

BEGIN { fname = "/dev/null" }
/<header>/,/<\/header>/ { hdr = hdr $0 "\n"; next }
/^<event / {
    events++
    if(events % 10000 == 1) {
        if(files++) close(fname)
        fname = sprintf("file%02d.txt", files)
        print hdr >fname
    }        
}            
{ print >>fname }

To run it: write it to a file script.awk, then run:

gawk -f script.awk file.txt
  • I am not sure how to run it. I assume this should be in a script? I have add a bigger file to my questions - this should be enough to make a test. – greyxray Oct 2 '16 at 19:48
  • Yes, you need to write it to a file. Answer updated. – Satō Katsura Oct 2 '16 at 19:55
  • thanks for clarification. However if I run it - I get no error message and no new files. Any ideas? – greyxray Oct 2 '16 at 20:08
  • Ok, it should actually work now. – Satō Katsura Oct 2 '16 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.