3

I have a huge file to process and I could not manage to get exactly what I need. Please note that I do not know before hand how many times this occurs in one file (e.g., could happen > 1000 times per files).

Below is my input file (TAB delimited), where $1 is the line number. The broken sequence of numbers occurs at $3:

797  47 M797    1     365.0     0.05     0.05 A 0.825
798  47 M798    1     365.0     0.05     0.05 A 0.825
799  47 M799    1     365.0     0.70     0.70 A 0.404
800  47 M800    1     365.0     0.00     0.00 A 0.990
801  47 M802    1     365.0     0.29     0.29 A 0.591
802  47 M803    1     365.0     0.12     0.12 A 0.726

This is what I want:

797  47 M797    1     365.0     0.05     0.05 A 0.825
798  47 M798    1     365.0     0.05     0.05 A 0.825
799  47 M799    1     365.0     0.70     0.70 A 0.404
800  47 M800    1     365.0     0.00     0.00 A 0.990
801  
802  47 M802    1     365.0     0.29     0.29 A 0.591
803  47 M803    1     365.0     0.12     0.12 A 0.726

This is the code I manage to write so far (filename is test.sh):

awk '
   marker=substr($3,2,6)
   { if (FNR < marker) {printf "\n"}
    }' ${1}

This is the output I got so far:

797       47 M797   1     365.0     0.05     0.05 A 0.825
798       47 M798   1     365.0     0.05     0.05 A 0.825
799       47 M799   1     365.0     0.70     0.70 A 0.404
800       47 M800   1     365.0     0.00     0.00 A 0.990
801       47 M802   1     365.0     0.29     0.29 A 0.591

802       47 M803    1    365.0     0.12     0.12 A 0.726

803       47 M804    1    365.0     0.08     0.08 A 0.777

If anyone has a better solution for this, please let me know.

2

You are tagged as awk, hopefully Python will be useful.

Code:

# !/usr/bin/python
import sys

def print_fixed_sequence(filename, line_num=0):
    with open(filename, 'rU') as f:
        for line in (x.strip() for x in f):
            _, f1, f2, data = line.split('\t', 3)
            rec_num = int(f2[1:])
            while line_num != rec_num:
                print(line_num)
                line_num += 1
            print('\t'.join((str(line_num), f1, f2, data)))
            line_num += 1

print_fixed_sequence(sys.argv[1], line_num=795)

Results:

795
796
797 47  M797    1   365.0   0.05    0.05    A   0.825
798 47  M798    1   365.0   0.05    0.05    A   0.825
799 47  M799    1   365.0   0.70    0.70    A   0.404
800 47  M800    1   365.0   0.00    0.00    A   0.990
801
802 47  M802    1   365.0   0.29    0.29    A   0.591
803 47  M803    1   365.0   0.12    0.12    A   0.726
| improve this answer | |
  • Thanks for your fast replyStephen, but python is giving me following errors: Traceback (most recent call last):File "test.py", line 15, in <module> print_fixed_sequence(sys.argv[1], line_num=795) File "test.py", line 8, in print_fixed_sequence rec_num = int(f2[1:]) ValueError: invalid literal for int() with base 10: '' – Newby Jun 28 '17 at 21:44
  • So f2[1:] takes everything after the first letter of the third column as the number to turn into the line number. Is the format of the file Mxxx? Or does it need to skip more than one letter? – Stephen Rauch Jun 28 '17 at 21:56
  • I think I know why this happened: in one of the fields, I have a few spaces instead of tabs. I changed the file and it worked fine. Thank you so much!!!! – Newby Jun 28 '17 at 22:02
2

Input file

797     47      M797    1       365.0   0.05    0.05    A       0.825
798     47      M798    1       365.0   0.05    0.05    A       0.825
799     47      M799    1       365.0   0.70    0.70    A       0.404
800     47      M800    1       365.0   0.00    0.00    A       0.990
801     47      M802    1       365.0   0.29    0.29    A       0.591
802     47      M804    1       365.0   0.12    0.12    A       0.726
803     47      M807    1       365.0   0.12    0.12    A       0.726
804     47      M808    1       365.0   0.12    0.12    A       0.726
805     47      M812    1       365.0   0.12    0.12    A       0.726
806     47      M813    1       365.0   0.12    0.12    A       0.726

Code

#!/usr/bin/awk -f
BEGIN{ al=790               # added lines at start
       ap=0                 # appended lines (missing)
     }
{
    lc=$1                # line counter
    $1=lc+ap             # adjust line number
    mr=substr($3,2,6)    # marker
    while( NR+al+ap < lc ){ print "added ++" ++al, ap,"lc=" lc}
    while( NR+al+ap < mr ){ print $1; $1=lc+(++ap); }
    if   ( lc+ap   == mr ){ print }
    if   ( lc+ap    > mr ){ print "errorerror"; exit}
}

Output

added ++791 0 lc=797
added ++792 0 lc=797
added ++793 0 lc=797
added ++794 0 lc=797
added ++795 0 lc=797
added ++796 0 lc=797
797 47 M797 1 365.0 0.05 0.05 A 0.825
798 47 M798 1 365.0 0.05 0.05 A 0.825
799 47 M799 1 365.0 0.70 0.70 A 0.404
800 47 M800 1 365.0 0.00 0.00 A 0.990
801
802 47 M802 1 365.0 0.29 0.29 A 0.591
803
804 47 M804 1 365.0 0.12 0.12 A 0.726
805
806
807 47 M807 1 365.0 0.12 0.12 A 0.726
808 47 M808 1 365.0 0.12 0.12 A 0.726
809
810
811
812 47 M812 1 365.0 0.12 0.12 A 0.726
813 47 M813 1 365.0 0.12 0.12 A 0.726
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.