0

I have a large file that I am trying to split into columns based on the values of the second column. I would like a series of output files that contains the data at the 'switching point'. The switching point is irregular, so I can't split by row number. The data below gives a small extract of the ~0.5M rows

...
605.144302472158    -5.18798828125E-07  0.98388671875
606.144302469633    -5.18798828125E-07  0.98419189453125
606.644327468370    -0.000150299072265625   0.93475341796875
... (~900-1800 rows)
1505.64432519861    -0.000150115966796875   0.4075927734375
1506.14432519735    -0.000150177001953125   0.40753173828125
1507.14435219483    -3.60107421875006E-07   0.41827392578125
1508.14435219230    -3.47900390625003E-07   0.420135498046875
... 

and the output should look like this: file 1

...
    605.144302472158    -5.18798828125E-07  0.98388671875
    606.144302469633    -5.18798828125E-07  0.98419189453125

file 2

606.644327468370    -0.000150299072265625   0.93475341796875
... (900-1800 rows)
1505.64432519861    -0.000150115966796875   0.4075927734375
1506.14432519735    -0.000150177001953125   0.40753173828125

file 3

1507.14435219483    -3.60107421875006E-07   0.41827392578125
1508.14435219230    -3.47900390625003E-07   0.420135498046875
... 

using awk '$2>-0.000001 {print}' input.txt

and awk '$2<-0.000001 {print}' input.txt

I can select the right data, but I don't know how to generate a new file each time $2>-0.000001 switches from 'true' to 'false'.

Using awk '{print > (++a[$2>-0.000001] ".txt")}' input.txt gets me a step closer by printing the output to files, but prints individul files for each row where the condition is met.

I guess I need some sort of while loop, but I'm not sure what it's supposed to look like.

0

To switch the output to a new file every time a certain criterium is met, you can use a variable storing the name of the output file, and change it every time you see a match for that datum:

$ cat input
17
1
4
13
9
10
7
5
18
19
3
6
16
8
12
15
14
2
11
20
$ awk 'BEGIN { outfilenum=1; outfile="file"outfilenum} $1 > 10 { outfilenum++; outfile = "file"outfilenum} {print >> outfile}' input
$ paste file*
11  20  17  13  18  19  16  12  15  14
        1   9       3   8           2
        4   10      6
            7
            5

To make the matched line appear at the end of each file, simply move the check to after the print routine, rather than before it as shown here.

  • Thank you. This works, but I notice that it generates a new file for every valid row. Is there a command to 'clean up' the files that are no longer necessary? When I tried to select and delete 40000 files my PC restarted itself. – Frederik Huld May 9 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.