How can I use basename with parallel?

Question

I have files like this on a Linux system:

    10S1_S5_L002_chrm.fasta  SRR3184711_chrm.fasta    SRR3987378_chrm.fasta  SRR4029368_chrm.fasta  SRR5204465_chrm.fasta    SRR5997546_chrm.fasta
13_S7_L003_chrm.fasta    SRR3184712_chrm.fasta    SRR3987379_chrm.fasta  SRR4029369_chrm.fasta  SRR5204520_chrm.fasta    SRR5997547_chrm.fasta
14_S8_L003_chrm.fasta    SRR3184713_chrm.fasta    SRR3987380_chrm.fasta  SRR4029370_chrm.fasta  SRR5208699_chrm.fasta    SRR5997548_chrm.fasta
17_S4_L002_chrm.fasta    SRR3184714_chrm.fasta    SRR3987415_chrm.fasta  SRR4029371_chrm.fasta  SRR5208700_chrm.fasta    SRR5997549_chrm.fasta
3_S1_L001_chrm.fasta     SRR3184715_chrm.fasta    SRR3987433_chrm.fasta  SRR4029372_chrm.fasta  SRR5208701_chrm.fasta    SRR5997550_chrm.fasta
4_S2_L001_chrm.fasta     SRR3184716_chrm.fasta    SRR3987482_chrm.fasta  SRR4029373_chrm.fasta  SRR5208770_chrm.fasta    SRR5997551_chrm.fasta
50m_S10_L004_chrm.fasta  SRR3184717_chrm.fasta    SRR3987489_chrm.fasta  SRR4029374_chrm.fasta  SRR5208886_chrm.fasta    SRR5997552_chrm.fasta
5_S3_L001_chrm.fasta     SRR3184718_chrm.fasta    SRR3987493_chrm.fasta  SRR4029375_chrm.fasta  SRR5211153_chrm.fasta    SRR6050903_chrm.fasta
65m_S11_L005_chrm.fasta  SRR3184719_chrm.fasta    SRR3987495_chrm.fasta  SRR4029376_chrm.fasta  SRR5211162_chrm.fasta    SRR6050905_chrm.fasta
6_S6_L002_chrm.fasta     SRR3184720_chrm.fasta    SRR3987647_chrm.fasta  SRR4029377_chrm.fasta  SRR5211163_chrm.fasta    SRR6050920_chrm.fasta
70m_S12_L006_chrm.fasta  SRR3184721_chrm.fasta    SRR3987651_chrm.fasta  SRR4029378_chrm.fasta  SRR5215118_chrm.fasta    SRR6050921_chrm.fasta
80m_S1_L002_chrm.fasta   SRR3184722_chrm.fasta    SRR3987657_chrm.fasta  SRR4029379_chrm.fasta  SRR5247122_chrm.fasta    SRR6050958_chrm.fasta

In all there are 423 I was asked to cut them in 32 parts for an optimal parallelisation on 32 CPU So now I have this:

    10S1_S5_L002_chrm.part-10.fasta  SRR3986254_chrm.part-26.fasta  SRR4029372_chrm.part-22.fasta    SRR5581526-1_chrm.part-20.fasta
10S1_S5_L002_chrm.part-11.fasta  SRR3986254_chrm.part-27.fasta  SRR4029372_chrm.part-23.fasta    SRR5581526-1_chrm.part-21.fasta
10S1_S5_L002_chrm.part-12.fasta  SRR3986254_chrm.part-28.fasta  SRR4029372_chrm.part-24.fasta    SRR5581526-1_chrm.part-22.fasta
10S1_S5_L002_chrm.part-13.fasta  SRR3986254_chrm.part-29.fasta  SRR4029372_chrm.part-25.fasta    SRR5581526-1_chrm.part-23.fasta
10S1_S5_L002_chrm.part-14.fasta  SRR3986254_chrm.part-2.fasta   SRR4029372_chrm.part-26.fasta    SRR5581526-1_chrm.part-24.fasta
10S1_S5_L002_chrm.part-15.fasta  SRR3986254_chrm.part-30.fasta  SRR4029372_chrm.part-27.fasta    SRR5581526-1_chrm.part-25.fasta
10S1_S5_L002_chrm.part-16.fasta  SRR3986254_chrm.part-31.fasta  SRR4029372_chrm.part-28.fasta    SRR5581526-1_chrm.part-26.fasta
10S1_S5_L002_chrm.part-17.fasta  SRR3986254_chrm.part-32.fasta  SRR4029372_chrm.part-29.fasta    SRR5581526-1_chrm.part-27.fasta
10S1_S5_L002_chrm.part-18.fasta  SRR3986254_chrm.part-3.fasta   SRR4029372_chrm.part-2.fasta     SRR5581526-1_chrm.part-28.fasta
10S1_S5_L002_chrm.part-19.fasta  SRR3986254_chrm.part-4.fasta   SRR4029372_chrm.part-30.fasta    SRR5581526-1_chrm.part-29.fasta
10S1_S5_L002_chrm.part-1.fasta   SRR3986254_chrm.part-5.fasta   SRR4029372_chrm.part-3.fasta     SRR5581526-1_chrm.part-2.fasta
10S1_S5_L002_chrm.part-20.fasta  SRR3986254_chrm.part-6.fasta   SRR4029372_chrm.part-4.fasta     SRR5581526-1_chrm.part-30.fasta
10S1_S5_L002_chrm.part-21.fasta  SRR3986254_chrm.part-7.fasta   SRR4029372_chrm.part-5.fasta     SRR5581526-1_chrm.part-31.fasta

I want to apply a command from the CRISPRCasFinder tool The command works well when I use it alone on 1 namefile.fasta The command also works well when I use parallel on namefile.part*.fasta.

But when I try to make the command more general by using basename, nothing works. I want to use basename to keep the name of my input files in the output folder.

I tried this on a smaller data set:

time parallel 'dossierSortie=$(basename -s .fasta {}) ; singularity exec -B $PWD /usr/local/CRISPRCasFinder-release-4.2.20/CrisprCasFinder.simg perl /usr/local/CRISPRCasFinder/CRISPRCasFinder.pl -so /usr/local/CRISPRCasFinder/sel392v2.so -cf /usr/local/CRISPRCasFinder/CasFinder-2.0.3 -drpt /usr/local/CRISPRCasFinder/supplementary_files/repeatDirection.tsv -rpts /usr/local/CRISPRCasFinder/supplementary_files/Repeat_List.csv -cas -def G --meta -out /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/Result{} -in /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/{}' ::: *_chrm.part*.fasta

And it did this

    ERR358546_chrm.part-1.fasta    SRR4029114_k141_23527.fna.bck   SRR5100341_k141_10416.fna.lcp   SRR5100345_k141_3703.fna.al1
ERR358546_chrm.part-2.fasta    SRR4029114_k141_23527.fna.bwt   SRR5100341_k141_10416.fna.llv   SRR5100345_k141_3703.fna.bck
ERR358546_chrm.part-3.fasta    SRR4029114_k141_23527.fna.des   SRR5100341_k141_10416.fna.ois   SRR5100345_k141_3703.fna.bwt
ERR358546_chrm.part-4.fasta    SRR4029114_k141_23527.fna.lcp   SRR5100341_k141_10416.fna.prj   SRR5100345_k141_3703.fna.des
ERR358546_chrm.part-5.fasta    SRR4029114_k141_23527.fna.llv   SRR5100341_k141_10416.fna.sds   SRR5100345_k141_3703.fna.lcp
ERR358546_chrm.part-6.fasta    SRR4029114_k141_23527.fna.ois   SRR5100341_k141_10416.fna.sti1  SRR5100345_k141_3703.fna.llv
ERR358546_k141_26987.fna       SRR4029114_k141_23527.fna.prj   SRR5100341_k141_10416.fna.suf   SRR5100345_k141_3703.fna.ois
ERR358546_k141_33604.fna       SRR4029114_k141_23527.fna.sds   SRR5100341_k141_10416.fna.tis   SRR5100345_k141_3703.fna.prj
ERR358546_k141_90631.fna       SRR4029114_k141_23527.fna.sti1  SRR5100341_k141_10942.fna       SRR5100345_k141_3703.fna.sds
ResultERR358546_chrm.part-3    SRR4029114_k141_23527.fna.suf   SRR5100341_k141_164.fna         SRR5100345_k141_3703.fna.sti1
ResultERR358546_chrm.part-4    SRR4029114_k141_23527.fna.tis   SRR5100341_k141_3046.fna        SRR5100345_k141_3703.fna.suf
ResultSRR4029114_chrm.part-1   SRR5100341_chrm.part-10.fasta   SRR5100341_k141_3968.fna        SRR5100345_k141_3703.fna.tis
ResultSRR4029114_chrm.part-4   SRR5100341_chrm.part-11.fasta   SRR5100341_k141_631.fna         SRR5100345_k141_4429.fna
ResultSRR5100341_chrm.part-10  SRR5100341_chrm.part-12.fasta   SRR5100341_k141_6376.fna        SRR5100345_k141_4832.fna
ResultSRR5100341_chrm.part-11  SRR5100341_chrm.part-13.fasta   SRR5100341_k141_8699.fna        SRR5100345_k141_6139.fna
ResultSRR5100341_chrm.part-3   SRR5100341_chrm.part-1.fasta    SRR5100341_k141_8892.fna        SRR5100345_k141_731.fna
ResultSRR5100341_chrm.part-9   SRR5100341_chrm.part-2.fasta    SRR5100345_chrm.part-10.fasta   SRR5100345_k141_731.fna.al1
ResultSRR5100345_chrm.part-1   SRR5100341_chrm.part-3.fasta    SRR5100345_chrm.part-1.fasta    SRR5100345_k141_731.fna.bck
ResultSRR5100345_chrm.part-4   SRR5100341_chrm.part-4.fasta    SRR5100345_chrm.part-2.fasta    SRR5100345_k141_731.fna.bwt
ResultSRR5100345_chrm.part-9   SRR5100341_chrm.part-5.fasta    SRR5100345_chrm.part-3.fasta    SRR5100345_k141_731.fna.des
SRR4029114_chrm.part-1.fasta   SRR5100341_chrm.part-6.fasta    SRR5100345_chrm.part-4.fasta    SRR5100345_k141_731.fna.lcp
SRR4029114_chrm.part-2.fasta   SRR5100341_chrm.part-7.fasta    SRR5100345_chrm.part-5.fasta    SRR5100345_k141_731.fna.llv
SRR4029114_chrm.part-3.fasta   SRR5100341_chrm.part-8.fasta    SRR5100345_chrm.part-6.fasta    SRR5100345_k141_731.fna.ois
SRR4029114_chrm.part-4.fasta   SRR5100341_chrm.part-9.fasta    SRR5100345_chrm.part-7.fasta    SRR5100345_k141_731.fna.prj
SRR4029114_chrm.part-5.fasta   SRR5100341_k141_10416.fna       SRR5100345_chrm.part-8.fasta    SRR5100345_k141_731.fna.sds
SRR4029114_k141_14384.fna      SRR5100341_k141_10416.fna.al1   SRR5100345_chrm.part-9.fasta    SRR5100345_k141_731.fna.sti1
SRR4029114_k141_16765.fna      SRR5100341_k141_10416.fna.bck   SRR5100345_k141_1211.fna        SRR5100345_k141_731.fna.suf
SRR4029114_k141_23527.fna      SRR5100341_k141_10416.fna.bwt   SRR5100345_k141_2884.fna        SRR5100345_k141_731.fna.tis
SRR4029114_k141_23527.fna.al1  SRR5100341_k141_10416.fna.des   SRR5100345_k141_3703.fna

The names of the folder are not okay because I want for example just ResultERR358546 and not ResultERR358546_chrm.part-2.fasta And I don't want a result for each part but only for each ID.

Answer 1

Your basename command only removes the fixed .fasta extension - as far as I know it cannot remove a variable pattern.

However GNU parallel provides a Perl expression replacement string facility that is much more powerful than basename - ex. given

$ ls *_chrm.part*.fasta
ERR358546_chrm.part-2.fasta  ERR358546_chrm.part-5.fasta  ERR358546_chrm.part-8.fasta
ERR358546_chrm.part-3.fasta  ERR358546_chrm.part-6.fasta  ERR358546_chrm.part-9.fasta
ERR358546_chrm.part-4.fasta  ERR358546_chrm.part-7.fasta

then

$ parallel echo Result'{= s:_.*$:: =}' ::: *_chrm.part*.fasta
ResultERR358546
ResultERR358546
ResultERR358546
ResultERR358546
ResultERR358546
ResultERR358546
ResultERR358546
ResultERR358546

where the substitution s:_.*$:: replaces everything after an underscore with nothing. Transplanting to your original command:

time parallel ' 
  singularity exec -B "$PWD" /usr/local/CRISPRCasFinder-release-4.2.20/CrisprCasFinder.simg \
  perl /usr/local/CRISPRCasFinder/CRISPRCasFinder.pl \
  -so /usr/local/CRISPRCasFinder/sel392v2.so \
  -cf /usr/local/CRISPRCasFinder/CasFinder-2.0.3 \
  -drpt /usr/local/CRISPRCasFinder/supplementary_files/repeatDirection.tsv \
  -rpts /usr/local/CRISPRCasFinder/supplementary_files/Repeat_List.csv \
  -cas -def G --meta \
  -out /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/Result'{= s:_.*$:: =}' \
  -in /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/{}
' ::: *_chrm.part*.fasta

---

If you want to capture and include the part index, you could modify the expression to

Result'{= s:_chrm\.part-(\d+)\.fasta$:_$1: =}'

or

'{= s:_chrm\.part-(\d+)\.fasta$:Result_$1: =}'

for example.

Answer 2

Your basename command only removes the .fasta:

$ basename ERR358546_chrm.part-1.fasta .fasta
ERR358546_chrm.part-1

To get only the sequence name, you need to remove more and since the number after part changes, you can't use basename. You can, however, use the shell itself and remove everything after the first _:

$ name="ERR358546_chrm.part-1.fasta"
$ newName="${name%%_*}"
$ echo "$newName"
ERR358546

Next, you need to actually use this. You have dossierSortie=$(basename -s .fasta {}) but you never actually use $dossierSortie. Since you are using parallel, which will replace {} with each of the input file names, you need to first store that as a shell variable and then apply the replacement above:

parallel ... 'name="{}"; dossierSortie="${name%%_*}"

So, this should do what you want:

time parallel 'name={}; dossierSortie="${name%%_*}"; \
     singularity exec -B "$PWD" \
    /usr/local/CRISPRCasFinder-release-4.2.20/CrisprCasFinder.simg perl /usr/local/CRISPRCasFinder/CRISPRCasFinder.pl \
    -so /usr/local/CRISPRCasFinder/sel392v2.so \
    -cf /usr/local/CRISPRCasFinder/CasFinder-2.0.3 \
    -drpt /usr/local/CRISPRCasFinder/supplementary_files/repeatDirection.tsv \
    -rpts /usr/local/CRISPRCasFinder/supplementary_files/Repeat_List.csv \
    -cas -def G --meta \
    -out /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/Result"${dossierSortie}" \
    -in /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/{}' ::: *_chrm.part*.fasta

Or, in case the line breaks trip you up, on a single line:

time parallel 'name={}; dossierSortie="${name%%_*}"; singularity exec -B "$PWD" /usr/local/CRISPRCasFinder-release-4.2.20/CrisprCasFinder.simg perl /usr/local/CRISPRCasFinder/CRISPRCasFinder.pl -so /usr/local/CRISPRCasFinder/sel392v2.so -cf /usr/local/CRISPRCasFinder/CasFinder-2.0.3 -drpt /usr/local/CRISPRCasFinder/supplementary_files/repeatDirection.tsv -rpts /usr/local/CRISPRCasFinder/supplementary_files/Repeat_List.csv -cas -def G --meta -out /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/Result"${dossierSortie}" -in /databis/defontis/Dossier_fasta_chrm_avec_CRISPRCasFinder/Test/{}' ::: *_chrm.part*.fasta