1

I will like to grep a particular pattern from 244 files and generate respective output files for each of them in a single command, how do I do that?

My command for a single file is as follows:

grep -v '@SQ' *.sam | grep -v '@HD' cut -f 3 | sort | uniq -c | sort -nrk1 > output_count_file.txt

Instead of doing this 244 times for each individual file, how do I do it at once?

Examples of my input files are:

A1_001.fastq.sam
A2_001.fastq.sam
B6_001.fastq.sam

And I would like to grep some information from these input files into their respective output files e.g.:

A1_001.txt
A2_001.txt
B6_001.txt

From these output files, I need to grep again to combine information from the output files to another file.

If I do grep for one of these files e.g. A1_001.fastq.sam, I will get the following information:

33 chr20:4804587-4804609__hsa_VP64_wgcod_3_27753
33 chr13:113242648-113242670__hsa_VP64_wgcod_2_46197
32 chr8:144718034-144718056__hsa_VP64_wgcod_2_48778
30 chr6:24126264-24126286__hsa_VP64_wgcod_1_71312

I need to identify gene names from the second column (e.g. chr20:4804587-4804609) by comparing to a reference file which has the following information:

GTGCGCAGCGCTGAGTGTCG    YBEY    NM_001006114_utr5_0_0_chr21_47706267_f;NM_058181_utr5_0_0_chr21_47706267_f  chr21   47706143    47706165
AGCAGGCGGACAGTAGGACG    AUP1    NM_181575_utr5_11_0_chr2_74756757_r chr2    74757053    74757075
TAGGGGCAATGAATGGCGAG    APEX2   NM_001271748_utr5_0_0_chrX_55026756_f;NM_014481_utr5_0_0_chrX_55026756_f    chrX    55026610    55026632

Hope my question is not too complicated.

9
  • 3
    What are you using your output files for? You can just grep all the files directly at the same time. Which your command already does (*.sam). And why are you using multiple "grep" commands in a pipeline? What is your real desired input and output? There are probably much easier approaches but it's hard to say when you don't provide sample input and output.
    – Wildcard
    Jun 22, 2017 at 4:00
  • Given recent GNU awk try awk -F'\t' 'BEGIN{PROCINFO["sorted_in"]="@val_num_desc"} !/@SQ/&&!/@HD/{n[$3]++} ENDFILE{for(i in n)print n[i],i>FILENAME".out"; delete n}' *.sam Jun 22, 2017 at 5:57
  • You can also use a for loop, google "examples bash for loop files".
    – thecarpy
    Jun 22, 2017 at 7:04
  • Thanks guys! I edited my question, hopefully it is clearer now? I want to count some lines from my 244 input files and output them into individual .txt files.
    – user237179
    Jun 22, 2017 at 20:40
  • 1
    You are right @Wildcard! I need the output .txt files for downstream processing. I edited my question again. I am a beginner in coding so apparently I can't even ask question properly, please bear with me. If you understand my question and can help me to write a script for that it will be very helpful! As I do not know how to do the step two grep too...
    – user237179
    Jun 22, 2017 at 22:46

2 Answers 2

1

You can achieve that by using find with the -exec flag. This will go through all of the files in the directory, and execute your grep individually on each file. You can put a placeholder for the output file to distinguish them.

The command would look something like this:

$ find . -iname "*.sam" -exec grep -v '@SQ' {} | grep -v '@HD' cut -f 3 | sort | uniq -c | sort -nrk1 > {}_output_count_file.txt \;

Note: I did not test this, so you will probably need to fix some funkiness with the escapes and placeholders, but it's a start.

0

See if you can make a function that can process a single file:

doit() {
  sam=$1
  grep -v '@SQ' "$sam".sam |
    grep -v '@HD' |
    cut -f 3 |
    sort |
    uniq -c |
    sort -nrk1 > "$sam"_count.txt
}
export -f doit

parallel doit {.} ::: *.sam

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .