extract everything between character

Question

I have multiple text files, e.g.,

sample1.tsv

## index of replication (iRep) - thresholds: min cov. = 5, min wins. = 0.98, min r^2 = 0.9, max fragments/Mbp = 175, GC correction min r^2 = 0.0    
# genome    ./bam/18097D-02-01.sam.sorted.sam
18097D-02-01_bin.11.fa  1.295179372
18097D-02-01_bin.13.fa  1.284880274
18097D-02-01_bin.15.fa  1.339609918
#   
## un-filtered index of replication (iRep)  
# genome    ./bam/18097D-02-01.sam.sorted.sam
18097D-02-01_bin.11.fa  1.295179372
18097D-02-01_bin.13.fa  1.284880274
18097D-02-01_bin.15.fa  1.339609918
#   
## raw index of replication (no GC bias correction) 
# genome    ./bam/18097D-02-01.sam.sorted.sam
18097D-02-01_bin.11.fa  1.298934455
18097D-02-01_bin.13.fa  1.2885746
#

sample2.tsv

## index of replication (iRep) - thresholds: min cov. = 5, min wins. = 0.98, min r^2 = 0.9, max fragments/Mbp = 175, GC correction min r^2 = 0.0    
# genome    ./bam/18097D-02-02.sam.sorted.sam
18097D-02-02_bin.11.fa  1.59665286
18097D-02-02_bin.13.fa  1.332990306
18097D-02-02_bin.14.fa  1.499196606
18097D-02-02_bin.6.fa   1.323465715
18097D-02-02_bin.9.fa   1.583302299
#   
## un-filtered index of replication (iRep)  
# genome    ./bam/18097D-02-02.sam.sorted.sam
18097D-02-02_bin.11.fa  1.59665286
18097D-02-02_bin.13.fa  1.332990306
18097D-02-02_bin.14.fa  1.499196606
18097D-02-02_bin.6.fa   1.323465715
18097D-02-02_bin.9.fa   1.583302299
#   
## raw index of replication (no GC bias correction) 
# genome    ./bam/18097D-02-02.sam.sorted.sam
18097D-02-02_bin.11.fa  1.603339021
18097D-02-02_bin.13.fa  1.366124796
18097D-02-02_bin.14.fa  1.502052999
18097D-02-02_bin.6.fa   1.324573575
18097D-02-02_bin.9.fa   1.618136032
#   

I want to extract everything from all the files from first ## to second ##, that means I want output like-

## index of replication (iRep) - thresholds: min cov. = 5, min wins. = 0.98, min r^2 = 0.9, max fragments/Mbp = 175, GC correction min r^2 = 0.0    
# genome    ./bam/18097D-02-01.sam.sorted.sam
**18097D-02-01_bin.11.fa 1.295179372
18097D-02-01_bin.13.fa 1.284880274
18097D-02-01_bin.15.fa 1.339609918**
#   
## un-filtered index of replication (iRep)
## index of replication (iRep) - thresholds: min cov. = 5, min wins. = 0.98, min r^2 = 0.9, max fragments/Mbp = 175, GC correction min r^2 = 0.0    
# genome    ./bam/18097D-02-02.sam.sorted.sam
**18097D-02-02_bin.11.fa 1.59665286
18097D-02-02_bin.13.fa 1.332990306
18097D-02-02_bin.14.fa 1.499196606
18097D-02-02_bin.6.fa  1.323465715
18097D-02-02_bin.9.fa  1.583302299**
#   
## un-filtered index of replication (iRep)

I was trying to do this, but no ouptput- sed -n '/##=/{s/.##=//;s/\S=.*//;p}' *.tsv > ../test.tsv

I actually want-

18097D-02-01_bin.11.fa 1.295179372
18097D-02-01_bin.13.fa 1.284880274
18097D-02-01_bin.15.fa 1.339609918

18097D-02-02_bin.11.fa 1.59665286
18097D-02-02_bin.13.fa 1.332990306
18097D-02-02_bin.14.fa 1.499196606
18097D-02-02_bin.6.fa  1.323465715
18097D-02-02_bin.9.fa  1.583302299

Thanks

Answer 1

#!/bin/bash
sample_dir="/C/Users/testuser/Desktop/sample" 
out_file="/C/Users/testuser/Desktop/sample/output.tsv"

for file in "$sample_dir"/*
do
   count=0
   while read line; do
        if [[ "$line" == "##"* ]] || [[ "$line" == "#"* ]]; then
           count=$((count+1))
           if [[ count -ge 4 ]]; then
              echo -e "" >> $out_file            
              continue 2
           fi
            continue 
        else
           echo -e "$line" >> $out_file
        fi
    done < $file
done

Answer 2

Use awk:

awk 'FNR==1{p=0}
     FNR==1 && NR>1 {print ""}
     $0 ~ /^##/ {p++}
     p==1 && $0 !~ /^#/
' sample*

Output:

18097D-02-01_bin.11.fa  1.295179372
18097D-02-01_bin.13.fa  1.284880274
18097D-02-01_bin.15.fa  1.339609918

18097D-02-02_bin.11.fa  1.59665286
18097D-02-02_bin.13.fa  1.332990306
18097D-02-02_bin.14.fa  1.499196606
18097D-02-02_bin.6.fa   1.323465715
18097D-02-02_bin.9.fa   1.583302299

Explanation:

• FNR==1{p=0} set pointer to 0 for each new file (FNR is the line number of the current file)
• FNR==1 && NR>1 {print ""} Print empty line for every file except the first
• $0 ~ /^##/ {p++} When line begins with ##, increase the pointer
• p==1 && $0 !~ /^#/ If pointer is 1 (which is the case from the first ## to the second ##, and the line does not begin with a #, print it.