3

I have a text file which has multiple columns in this manner:

 1 102.0184 109.5487 107.3716
 2 100.2430 107.5874 108.7558
 3 103.9029 105.7406 106.9046
 4 102.7495 108.1275 104.4731
 5 102.8825 105.6664 107.2141
 6 104.5323 108.8850 106.0267
 7 103.0479 107.3056 105.5108
 8 101.2433 108.6113 107.2551
 9 104.4821 108.4339 105.9864
 10 101.5879 106.3859 102.825

Then I filter it in the following way, let's say, for column two.

awk '$2<104 {print $1,$2}' file.txt 

Obtaining this:

1   102.0184
2   100.2430
3   103.9029
4   102.7495
5   102.8825
7   103.0479
8   101.2433
10  101.5879

I want to print in a different file for all the continuous ranges in the column one, so for this example I would have three output files:

file_1-5.tmp
1   102.0184
2   100.2430
3   103.9029
4   102.7495
5   102.8825
file_7-8.tmp
7   103.0479
8   101.2433
file_10.tmp
10  101.5879

How can I do that? How can I extend this routine for all the columns in my file? I would like to have a general solution to this problem (not dependent of the particular ranges given in this example) because I want to apply it to multiple files.

  • If you expect an answer to that you really should give an example of what the result should look like. – Hauke Laging Aug 14 '17 at 18:21
  • Please clarify the continuous range part. With your example, would you have have just file_1-10.tmp? Or several files? – xhienne Aug 14 '17 at 18:24
  • I expanded the output part for clarity. Thanks for the suggestion. – webbynator Aug 14 '17 at 18:28
2

The one liner awk command:

awk -v P=-1 '$2<104 {if ($1>P+1)N=$1;P=$1;print $1,$2>"file"N".tmp"}'

Giving result (if you don't mind files name)

==> file1.tmp <==
1   102.0184
2   100.2430
3   103.9029
4   102.7495
5   102.8825

==> file7.tmp <==
7 103.0479
8 101.2433

==> file10.tmp <==
10 101.5879
1

Below is an awk program. You can play with the column and the threshold given in arguments:

awk -v column=2 -v threshold=104 '
    function save() { if (lines != "") print lines >"file_" first "-" last ".txt" }

    ! ($column < threshold) {
        save()
        first = last = lines = ""
        next
    }

    { 
        if (first == "") first = $1
        last = $1
        lines = lines $1 OFS $column ORS
    }

    END { save() }
'

Note that continuous lines are kept in memory until they are saved. If you have hundreds of millions of continuous lines, this solution should be adapted (saving each line in a temporary file then renaming it when the last line of a continuous block is met).

  • @glennjackman Done. With OFS too ;-) – xhienne Aug 15 '17 at 1:16
0
awk '{$0 = $2 < 104 ? NR : RS}1' inputfile |
sed -Ee '
   $aq
   /./,/^$/!d;/./H;$ba;/^$/ba;d;:a
   g;s/.//;s/\n.*\n|\n/,/;x;s/.*//;x
   s/(.*),(.*)/&w file_\1_\2.tmp/
   /,/!s/.*/&w file_&.tmp/
' | ed -s inputfile -

Here we use the awk/sed/ed tools. Awk first generates line numbers for all thos lines that are less than 104. For others, it prints an empty line. Then Sed comes in and looks at line ranges from non-empty line to the next empty line. And stores up the line numbers in the hold. Now two types of ranges can be found: n,m or plain n. Using these we build up an ed command set which shall transform such ranges to: n,mw file_n_m.tmp and file_n.tmp respectively. Then ed promptly works on the input file using this dynamically generated ed script to get the lines into their .tmp files.

Here's one way to accomplish your task with Perl:

perl -lane '
   BEGIN { $fmt = sprintf q[%s%%s\n%s], (chr 39)x2; }
   if ( $F[1] < 104 ) {
      push @A, "@F[0,1]";
      if ( eof ) {
         my $f = join $", q<printf>, $fmt, map(qq[\"$_\"], @A), q[>], (( ! defined $a ) ? qq[file_${.}.tmp] : qq[file_${a}_${.}.tmp]);
         system("$f");
      } else { $a //= $.; }
   } else {
      next if ! defined $a;
      $b //= $.-1;
      my $f = join $", q<printf>, $fmt, map(qq[\"$_\"], @A), q[>], (( $a == $b ) ? qq[file_$b.tmp] : qq[file_${a}_$b.tmp]);
      system("$f");
      ($a, $b, @A) = (undef)x2;
   }
' yourfile

Results:

Given the input , the following 3 files are created: file_1_5.tmp file_7_8.tmp file_10.tmp with the contents

% more file_1_5.tmp file_7_8.tmp file_10.tmp

::::::::::::::
file_1_5.tmp
::::::::::::::
1 102.0184
2 100.2430
3 103.9029
4 102.7495
5 102.8825
::::::::::::::
file_7_8.tmp
::::::::::::::
7 103.0479
8 101.2433
::::::::::::::
file_10.tmp
::::::::::::::
10 101.5879

Explanation:

First the basic top-level idea: We keep an eye on wheter the 2nd field is falling behind the numeric 104. In the scenario when it does, means we need to print the previous range. Just keep in mind that for unilength ranges, the filename is modified accordingly to reflect that.

The other case when we are in the process of accumulating the current range, ($F[1] < 104) then just keep in mind that while doing so if we hit eof then we need to print the range now.

P.S.: The system command is dynamically created by using a dynamically created format, it's data is the first and second fields, and finally the .tmp filename is created according to the range.

$a and $b are range begin/end line numbers. Their states will inform us to take the proper decisions.

  • 1
    Why on earth are you making a system call? Perl can do whatever printf can do. You do a lot of quoting tricks to protect the arguments for the printf command, and that really harms readability. – glenn jackman Aug 15 '17 at 1:06
  • @glennjackman The system call is there to write to a file. OTW, we need to open a filehandle and then close it. As for the printf quoting, actually, since it's a single quoted command line, so to insert the single quotes for printf, we need to escape them , hence I thought of bringing in char function to do that. – user218374 Aug 15 '17 at 2:43
  • as @glennjackman said, perl can printf. and it should be no surprise that perl can also write to a file. it can also open a file, read it in and write it back out with changes. perl can even open a pipe to ed and printf a series of commands to it. BTW, I've written many perl scripts that open a pipe to cat if run with --debug or a pipe to bash or whatever otherwise. – cas Aug 15 '17 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.