0

I am trying to find a way to write a script that executes a command on varying number of files in subdirectories. I have 72 subdirectories each with varying number of sample files (fastq.gz files) with the one containing the most having 20 sample files. I used the following command but it gives me alot of empty folders and files when the number of files is less than 20 in a subdirectory.

#!/bin/sh
TOPHAT_BINARY=/home/alex/tools/tophat-2.1.0.Linux_x86_64/tophat2
GENE_REFERENCE=/home/alex/toxo/ref/genes/ToxoDB-27_TgondiiME49.gff
BOWTIE_INDEX=/home/alex/toxo/ref/bwt/ToxoDB-27_TgondiiME49_Genome
P=10 #use 10 threads
for FILE_ID in {001..072}
do
 for SAMPLE_ID in {001..020}
 do
 $TOPHAT_BINARY -G $GENE_REFERENCE -p $P -o /home/alex/toxo/alignments/Nishi_${FILE_ID}/sample_${SAMPLE_ID} $BOWTIE_INDEX /home/junya/bioinfo/NGS/original/Nishikawa.cell/rename_and_link/Nishi_${FILE_ID}/*_${SAMPLE_ID}.fastq.gz &
 mv /home/alex/toxo/alignments/Nishi_${FILE_ID}/sample_${SAMPLE_ID}/accepted_hits.bam /home/alex/toxo/alignments/Nishi_${FILE_ID}/sample_${SAMPLE_ID}.bam
 done
done

how do i edit the script

for SAMPLE_ID in {001..020}

so that the command only applies to the specific number of files in a particular subdirectory without outputting empty files?

Some of the subdirectories contain 2, 3, 10, 16 etc number of files so I just did 20 so if a subdirectory has 2 sample files then the output will contain 18 empty files....I added a command to remove all empty files at the end and it worked but Is there any other way to specify only the files contained in multiple subdirectories?

## secure and fast version ###
find /home/alex/toxo/alignments/Nishi_${FILE_ID} -type f -empty -print0 | xargs -0 -I {} /bin/rm "{}"
  • You could test for the existence of the file beforehand perhaps? – Tim S. Apr 1 '16 at 4:52
  • Some of the subdirectories contain 2, 3, 10, 16 etc number of files so I just did 20 so if a subdirectory has 2 sample files then the output will contain 18 empty files....I added a command to remove all empty files at the end and it worked but Is there any other way to specify only the files contained in multiple subdirectories? – alex kiarie Apr 1 '16 at 4:57
  • please see my answer below – MelBurslan Apr 1 '16 at 4:59
0

It looks like Mel's answer has solved your problem; here's another variation. Instead of looping over a large range of SAMPLE_ID's (and what if you eventually end up with more than your expected 20?), I changed the loop so that it uses bash's filename expansion to loop over all and only the sample_* files that actually exist in /home/alex/toxo/alignments/Nishi_${FILE_ID}. In order to get your original value of SAMPLE_ID back, I use Parameter Expansion to retrieve the last three charcters from the SAMPLE_ID filename. Note that this breaks if you end up with more than 999 sample_* files!

I also changed your she-bang header line to explicitly call bash (via env) because if called as /bin/sh, bash will try to emulate Bourne shell, and lose features of the Bourne Again SHell (like fancy Parameter Expansion!).

As a side bonus, using SAMPLE_FILE in this way makes the commands a little bit shorter & easier to read!

#!/usr/bin/env bash
TOPHAT_BINARY=/home/alex/tools/tophat-2.1.0.Linux_x86_64/tophat2
GENE_REFERENCE=/home/alex/toxo/ref/genes/ToxoDB-27_TgondiiME49.gff
BOWTIE_INDEX=/home/alex/toxo/ref/bwt/ToxoDB-27_TgondiiME49_Genome
P=10 #use 10 threads
for FILE_ID in {001..003}
do
  for SAMPLE_FILE in /home/alex/toxo/alignments/Nishi_${FILE_ID}/sample_*
  do
    SAMPLE_ID=${SAMPLE_FILE: -3}
    $TOPHAT_BINARY -G $GENE_REFERENCE -p $P -o $SAMPLE_FILE $BOWTIE_INDEX /home/junya/bioinfo/NGS/original/Nishikawa.cell/rename_and_link/Nishi_${FILE_ID}/*_${SAMPLE_ID}.fastq.gz
    mv ${SAMPLE_FILE}/accepted_hits.bam ${SAMPLE_FILE}/sample_${SAMPLE_ID}.bam
    echo
  done
done
0

right after second do line, insert a new line and type

if [ -a /home/alex/toxo/alignments/Nishi_${FILE_ID}/sample_${SAMPLE_ID} ] ; then

and right before the first done line, again, insert a new line and type

fi

this will check the existence of the file before trying to process it. If it doesn't exist, it will skip over that SAMPLE_ID value.

  • That path is the output directory the input directory is /home/junya/bioinfo/NGS/original/Nishikawa.cell/rename_and_link/Nishi_${FILE_ID}. the problem is i have to create the output directories first so they are all empty. furthermore some will definitely remain empty after the run – alex kiarie Apr 1 '16 at 5:32
  • Why not use -e instead of -a? -e is specified in POSIX; -a is not. – Wildcard Apr 1 '16 at 5:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.