1

I have a 2Gb file. That has a a header and many-many "event"-structures. This is how the beginning will look:

<run example>
<header>
 5
   This is header                                                   
</header>

<event  =             22>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
       0       0       0       0       0       1       1       8
       0       1       0       2       1       5       2       0
       2       1       3       7       3       1       1       0
       1       0   10100       2       3       1       5       1
       1       5       1       7       2       3       2       2
</evhead>
 0 97
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0  4.63012694434E+00  2.62561831936E+00 -2.31855757639E+01  2.37887130977E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      22       1       0       0       0 -4.63012694434E+00 -2.62561831936E+00 -4.31442423592E+00  3.71128690719E+00 -5.75956188088E+00
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       2       0       0       0 -2.16995636615E-14 -1.11022302463E-15  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      22       4       0       0       0 -4.60626572550E+00 -2.61208727495E+00 -2.23619853289E+00  5.74815342040E+00  0.00000000000E+00
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
</event>

The whole file contains 97000 of such "event" blocks. So I want to split this file in 10 files in the way that each would contain the header AND 10 000 of the "event" blocks. All blocks differ by index (they are random). The last file will of course contain only 7 000 blocks.

I have tried multiple instructions from stack like here: https://stackoverflow.com/questions/8544197/splitting-a-file-in-linux-based-on-content https://stackoverflow.com/questions/8544197/splitting-a-file-in-linux-based-on-content but non did worked for me.

Below is a bit bigger example of the file for any tests (file_to_download):

<run example>
<header>
 5
   header                                                     
</header>

<event  =             22>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
</evhead>
 0 97
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
</event>

<event  =             26>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
</evhead>
 0 52
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
</event>

<event  =             31>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
       0       0       0       0       0       1       1       8
</evhead>
 0 92
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0  4.39003604933E+00  4.97037860337E+00 -2.04926313413E+01  2.15389187176E+01  5.10000000000E-04
</event>

<event  =             37>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
</evhead>
 0 77
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0  7.91768942174E+00  3.75815788575E+00 -2.09569980000E+01  2.27158385693E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
</event>

<event  =             41>
<evhead>
 8
       3       1       2       0       0       0       0       0
       0       0       1       0       1      30       0       1
       4       1       4       3       1       0       1       0
</evhead>
 0 122
           3      11       0       0       3       4  1.94791176123E-14  0.00000000000E+00 -2.75000000000E+01  2.75000000047E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3    2212       0       0       5       0 -1.94791176123E-14  0.00000000000E+00  9.20000000000E+02  9.20000478451E+02  9.38270000000E-01
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
           3      11       1       0       0       0 -3.63469912393E+00  3.95372353695E+00 -1.62133507727E+01  1.70796870892E+01  5.10000000000E-04
                                                      0.00000000000E+00  0.00000000000E+00  0.00000000000E+00  0.00000000000E+00
</event>
2
  • Is it an XML file? The snippet looks like one. Have you considering using an XML parser to split out this content? If not, is there any reason why the answer has to include awk? Oct 2, 2016 at 18:43
  • 1
    @roaima actually any native linux utility will do. I am using a computer where I am not allowed to instal any software
    – greyxray
    Oct 2, 2016 at 19:35

1 Answer 1

2

With GNU awk:

BEGIN { fname = "/dev/null" }
/<header>/,/<\/header>/ { hdr = hdr $0 "\n"; next }
/^<event / {
    events++
    if(events % 10000 == 1) {
        if(files++) close(fname)
        fname = sprintf("file%02d.txt", files)
        print hdr >fname
    }        
}            
{ print >>fname }

To run it: write it to a file script.awk, then run:

gawk -f script.awk file.txt
4
  • I am not sure how to run it. I assume this should be in a script? I have add a bigger file to my questions - this should be enough to make a test.
    – greyxray
    Oct 2, 2016 at 19:48
  • Yes, you need to write it to a file. Answer updated. Oct 2, 2016 at 19:55
  • thanks for clarification. However if I run it - I get no error message and no new files. Any ideas?
    – greyxray
    Oct 2, 2016 at 20:08
  • Ok, it should actually work now. Oct 2, 2016 at 20:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .