Split to multiple files based on first column when matched numeric ranges in another column with awk

Question

I have a text file which has multiple columns in this manner:

 1 102.0184 109.5487 107.3716
 2 100.2430 107.5874 108.7558
 3 103.9029 105.7406 106.9046
 4 102.7495 108.1275 104.4731
 5 102.8825 105.6664 107.2141
 6 104.5323 108.8850 106.0267
 7 103.0479 107.3056 105.5108
 8 101.2433 108.6113 107.2551
 9 104.4821 108.4339 105.9864
 10 101.5879 106.3859 102.825

Then I filter it in the following way, let's say, for column two.

awk '$2<104 {print $1,$2}' file.txt 

Obtaining this:

1   102.0184
2   100.2430
3   103.9029
4   102.7495
5   102.8825
7   103.0479
8   101.2433
10  101.5879

I want to print in a different file for all the continuous ranges in the column one, so for this example I would have three output files:

file_1-5.tmp
1   102.0184
2   100.2430
3   103.9029
4   102.7495
5   102.8825

file_7-8.tmp
7   103.0479
8   101.2433

file_10.tmp
10  101.5879

How can I do that? How can I extend this routine for all the columns in my file? I would like to have a general solution to this problem (not dependent of the particular ranges given in this example) because I want to apply it to multiple files.

Answer 1

The one liner awk command:

awk -v P=-1 '$2<104 {if ($1>P+1)N=$1;P=$1;print $1,$2>"file"N".tmp"}'

Giving result (if you don't mind files name)

==> file1.tmp <==
1   102.0184
2   100.2430
3   103.9029
4   102.7495
5   102.8825

==> file7.tmp <==
7 103.0479
8 101.2433

==> file10.tmp <==
10 101.5879

Answer 2

Below is an awk program. You can play with the column and the threshold given in arguments:

awk -v column=2 -v threshold=104 '
    function save() { if (lines != "") print lines >"file_" first "-" last ".txt" }

    ! ($column < threshold) {
        save()
        first = last = lines = ""
        next
    }

    { 
        if (first == "") first = $1
        last = $1
        lines = lines $1 OFS $column ORS
    }

    END { save() }
'

Note that continuous lines are kept in memory until they are saved. If you have hundreds of millions of continuous lines, this solution should be adapted (saving each line in a temporary file then renaming it when the last line of a continuous block is met).

Answer 3

awk '{$0 = $2 < 104 ? NR : RS}1' inputfile |
sed -Ee '
   $aq
   /./,/^$/!d;/./H;$ba;/^$/ba;d;:a
   g;s/.//;s/\n.*\n|\n/,/;x;s/.*//;x
   s/(.*),(.*)/&w file_\1_\2.tmp/
   /,/!s/.*/&w file_&.tmp/
' | ed -s inputfile -

Here we use the awk/sed/ed tools. Awk first generates line numbers for all thos lines that are less than 104. For others, it prints an empty line. Then Sed comes in and looks at line ranges from non-empty line to the next empty line. And stores up the line numbers in the hold. Now two types of ranges can be found: n,m or plain n. Using these we build up an ed command set which shall transform such ranges to: n,mw file_n_m.tmp and file_n.tmp respectively. Then ed promptly works on the input file using this dynamically generated ed script to get the lines into their .tmp files.

Here's one way to accomplish your task with Perl:

perl -lane '
   BEGIN { $fmt = sprintf q[%s%%s\n%s], (chr 39)x2; }
   if ( $F[1] < 104 ) {
      push @A, "@F[0,1]";
      if ( eof ) {
         my $f = join $", q<printf>, $fmt, map(qq[\"$_\"], @A), q[>], (( ! defined $a ) ? qq[file_${.}.tmp] : qq[file_${a}_${.}.tmp]);
         system("$f");
      } else { $a //= $.; }
   } else {
      next if ! defined $a;
      $b //= $.-1;
      my $f = join $", q<printf>, $fmt, map(qq[\"$_\"], @A), q[>], (( $a == $b ) ? qq[file_$b.tmp] : qq[file_${a}_$b.tmp]);
      system("$f");
      ($a, $b, @A) = (undef)x2;
   }
' yourfile

---

Results:

Given the input , the following 3 files are created: file_1_5.tmp file_7_8.tmp file_10.tmp with the contents

% more file_1_5.tmp file_7_8.tmp file_10.tmp

::::::::::::::
file_1_5.tmp
::::::::::::::
1 102.0184
2 100.2430
3 103.9029
4 102.7495
5 102.8825
::::::::::::::
file_7_8.tmp
::::::::::::::
7 103.0479
8 101.2433
::::::::::::::
file_10.tmp
::::::::::::::
10 101.5879

---

Explanation:

First the basic top-level idea: We keep an eye on wheter the 2nd field is falling behind the numeric 104. In the scenario when it does, means we need to print the previous range. Just keep in mind that for unilength ranges, the filename is modified accordingly to reflect that.

The other case when we are in the process of accumulating the current range, ($F[1] < 104) then just keep in mind that while doing so if we hit eof then we need to print the range now.

P.S.: The system command is dynamically created by using a dynamically created format, it's data is the first and second fields, and finally the .tmp filename is created according to the range.

$a and $b are range begin/end line numbers. Their states will inform us to take the proper decisions.