0

I will like to grep a particular pattern from 244 files and generate respective output files for each of them in a single command, how do I do that?

My command for a single file is as follows:

grep -v '@SQ' *.sam | grep -v '@HD' cut -f 3 | sort | uniq -c | sort -nrk1 > output_count_file.txt

Instead of doing this 244 times for each individual file, how do I do it at once?

Examples of my input files are:

A1_001.fastq.sam
A2_001.fastq.sam
B6_001.fastq.sam

And I would like to grep some information from these input files into their respective output files e.g.:

A1_001.txt
A2_001.txt
B6_001.txt

From these output files, I need to grep again to combine information from the output files to another file.

If I do grep for one of these files e.g. A1_001.fastq.sam, I will get the following information:

33 chr20:4804587-4804609__hsa_VP64_wgcod_3_27753
33 chr13:113242648-113242670__hsa_VP64_wgcod_2_46197
32 chr8:144718034-144718056__hsa_VP64_wgcod_2_48778
30 chr6:24126264-24126286__hsa_VP64_wgcod_1_71312

I need to identify gene names from the second column (e.g. chr20:4804587-4804609) by comparing to a reference file which has the following information:

GTGCGCAGCGCTGAGTGTCG    YBEY    NM_001006114_utr5_0_0_chr21_47706267_f;NM_058181_utr5_0_0_chr21_47706267_f  chr21   47706143    47706165
AGCAGGCGGACAGTAGGACG    AUP1    NM_181575_utr5_11_0_chr2_74756757_r chr2    74757053    74757075
TAGGGGCAATGAATGGCGAG    APEX2   NM_001271748_utr5_0_0_chrX_55026756_f;NM_014481_utr5_0_0_chrX_55026756_f    chrX    55026610    55026632

Hope my question is not too complicated.

  • 3
    What are you using your output files for? You can just grep all the files directly at the same time. Which your command already does (*.sam). And why are you using multiple "grep" commands in a pipeline? What is your real desired input and output? There are probably much easier approaches but it's hard to say when you don't provide sample input and output. – Wildcard Jun 22 '17 at 4:00
  • Given recent GNU awk try awk -F'\t' 'BEGIN{PROCINFO["sorted_in"]="@val_num_desc"} !/@SQ/&&!/@HD/{n[$3]++} ENDFILE{for(i in n)print n[i],i>FILENAME".out"; delete n}' *.sam – dave_thompson_085 Jun 22 '17 at 5:57
  • You can also use a for loop, google "examples bash for loop files". – thecarpy Jun 22 '17 at 7:04
  • Thanks guys! I edited my question, hopefully it is clearer now? I want to count some lines from my 244 input files and output them into individual .txt files. – user237179 Jun 22 '17 at 20:40
  • 1
    You are right @Wildcard! I need the output .txt files for downstream processing. I edited my question again. I am a beginner in coding so apparently I can't even ask question properly, please bear with me. If you understand my question and can help me to write a script for that it will be very helpful! As I do not know how to do the step two grep too... – user237179 Jun 22 '17 at 22:46
0

You can achieve that by using find with the -exec flag. This will go through all of the files in the directory, and execute your grep individually on each file. You can put a placeholder for the output file to distinguish them.

The command would look something like this:

$ find . -iname "*.sam" -exec grep -v '@SQ' {} | grep -v '@HD' cut -f 3 | sort | uniq -c | sort -nrk1 > {}_output_count_file.txt \;

Note: I did not test this, so you will probably need to fix some funkiness with the escapes and placeholders, but it's a start.

0

See if you can make a function that can process a single file:

doit() {
  sam=$1
  grep -v '@SQ' "$sam".sam |
    grep -v '@HD' |
    cut -f 3 |
    sort |
    uniq -c |
    sort -nrk1 > "$sam"_count.txt
}
export -f doit

parallel doit {.} ::: *.sam

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.