Grep multiple files and output to multiple files in a single command

Question

I will like to grep a particular pattern from 244 files and generate respective output files for each of them in a single command, how do I do that?

My command for a single file is as follows:

grep -v '@SQ' *.sam | grep -v '@HD' cut -f 3 | sort | uniq -c | sort -nrk1 > output_count_file.txt

Instead of doing this 244 times for each individual file, how do I do it at once?

Examples of my input files are:

A1_001.fastq.sam
A2_001.fastq.sam
B6_001.fastq.sam

And I would like to grep some information from these input files into their respective output files e.g.:

A1_001.txt
A2_001.txt
B6_001.txt

From these output files, I need to grep again to combine information from the output files to another file.

If I do grep for one of these files e.g. A1_001.fastq.sam, I will get the following information:

33 chr20:4804587-4804609__hsa_VP64_wgcod_3_27753
33 chr13:113242648-113242670__hsa_VP64_wgcod_2_46197
32 chr8:144718034-144718056__hsa_VP64_wgcod_2_48778
30 chr6:24126264-24126286__hsa_VP64_wgcod_1_71312

I need to identify gene names from the second column (e.g. chr20:4804587-4804609) by comparing to a reference file which has the following information:

GTGCGCAGCGCTGAGTGTCG    YBEY    NM_001006114_utr5_0_0_chr21_47706267_f;NM_058181_utr5_0_0_chr21_47706267_f  chr21   47706143    47706165
AGCAGGCGGACAGTAGGACG    AUP1    NM_181575_utr5_11_0_chr2_74756757_r chr2    74757053    74757075
TAGGGGCAATGAATGGCGAG    APEX2   NM_001271748_utr5_0_0_chrX_55026756_f;NM_014481_utr5_0_0_chrX_55026756_f    chrX    55026610    55026632

Hope my question is not too complicated.

Answer 1

You can achieve that by using find with the -exec flag. This will go through all of the files in the directory, and execute your grep individually on each file. You can put a placeholder for the output file to distinguish them.

The command would look something like this:

$ find . -iname "*.sam" -exec grep -v '@SQ' {} | grep -v '@HD' cut -f 3 | sort | uniq -c | sort -nrk1 > {}_output_count_file.txt \;

Note: I did not test this, so you will probably need to fix some funkiness with the escapes and placeholders, but it's a start.

Answer 2

See if you can make a function that can process a single file:

doit() {
  sam=$1
  grep -v '@SQ' "$sam".sam |
    grep -v '@HD' |
    cut -f 3 |
    sort |
    uniq -c |
    sort -nrk1 > "$sam"_count.txt
}
export -f doit

parallel doit {.} ::: *.sam