2

I am running the following bash script as is. I.e. sequentially. Note that the prefetch and fastq-dump are calls for downloading files from a database, and _shrink is a C++ program that I have made to shrink the fetched files (due to limited storage).

Is it possible for me to use threads for this? It takes ages to download the files.

Important! The commands inside each block must run sequentially, but the blocks are allowed to run in parallel. Moreover, I would like to only run 10 blocks at a time.

#!/bin/bash
g++ -std=c++11 shrink_files.cpp -o _shrink

#block1    
prefetch SRR837459
fastq-dump --fasta 0 SRR837459
rm ../../sra_sequences/sra/SRR837459.sra
./_shrink SRR837459
rm SRR837459.fasta
echo "SRR837459" >> list_of_done.txt

#block1    
prefetch SRR805782
fastq-dump --fasta 0 SRR805782
rm ../../sra_sequences/sra/SRR805782.sra
./_shrink SRR805782
rm SRR805782.fasta
echo "SRR805782" >> list_of_done.txt

#... more blocks

2 Answers 2

3

I think a nice way for parallel commands might be ... GNU parallel. Define your block as function as glenn jackmann suggested and then run them with parallel, where -j allows defining the maximal parallel runs. The advantage over glenn's approach is that if any block is done, a new one will be started so you always have 10 running at once. Note that you will have to export the function for parallel to be able to see it.

#!/bin/bash
g++ -std=c++11 shrink_files.cpp -o _shrink

block() {
  prefetch "$1"
  fastq-dump --fasta 0 "$1"
  ...
}

export -f block

parallel -j 10 block ::: id1 id2 id3 id4 ....

Replace idi with your ids. Alternatively if you have the IDs (SRR837459 and so on) in a file like

id1
id2
id3

then use this line instead (GNU Parallel reads from standard input if there is no other input source):

parallel -j 10 block

and run the script via

bash script < idlist.txt
3
  • What is the parallel -j 10 block ::: id1 id2 id3 id4 .... ? should it be in the script as well? Should I hardcode all id's in the script also? Btw. It gives me this error: parallel: command not found Mar 19, 2021 at 17:35
  • Yes, it goes to the script. parallel is a (n external) program that can be used to run similar command parallely. Like in your case you want the function block to run in parallel with different IDs (I called them id1, id2 as placeholder. Replace with: SRR837459, SRR837459 and so on). -j 10 limits the maximum number of active runs to 10. parallel seems not to be installed on your machine. Check your package manger, the homepage for a package or get the source code (installing is super simple - just look at the README and do as it says) - gnu.org/software/parallel .
    – FelixJN
    Mar 20, 2021 at 17:09
  • If you are not allowed to install software in your home dir, you can use parallel --embed on another machine that has GNU Parallel to generate a script that includes GNU Parallel - and just append FelixJN's lines to the script.
    – Ole Tange
    Mar 22, 2021 at 17:10
2

You can group commands to run in the background

untested

#!/bin/bash
g++ -std=c++11 shrink_files.cpp -o _shrink

block() {
    local id=$1
    prefetch "$id"
    fastq-dump --fasta 0 "$id"
    rm ../../sra_sequences/sra/"$id".sra
    ./_shrink "$id"
    rm "$id".fasta
    echo "$id" >> list_of_done.txt
}

# run first 10 blocks in the background
block SRR837459 &
block SRR805782 &
...
block 10th_id &

# and wait for them to complete
wait

# now start up on the next 10 ...
block 11th_id &
...

Or the following is a bit more programmatic:

# all the ids to fetch
ids=( SRR837459 SRR805782 ... )

while (( ${#ids[@]} > 0 )); do
    # copy the first 10 as positional parameters
    set -- "${ids[@]:0:10}"
    # launch the blocks
    for id; do
        block "$id" &
    done
    # wait until done
    wait
    # remove the first 10 from the list
    ids=( "${ids[@]:10}" )
done
4
  • To ensure that always 10 are running, one might count the parallel runs: active() {$(ps aux | grep '/bin/bash -c block' | grep -v grep | wc -l} and do something like i=0 ; nids=${#id[@]} ; while [ $i -lt $((nids-1)) ] ; do if [ $(active) -lt 10 ] ; then i=$((i+1)) ; block ${id[i]} & ; fi ; sleep 1 ; done (untested)
    – FelixJN
    Mar 18, 2021 at 21:56
  • When you write: "run in the background" what does it mean? I have tried with your first solution and it works. However, it does not work in the background. The command line is occupied and I cannot do anything but wait until finished. Mar 19, 2021 at 18:01
  • moreover, should I end with a & ? Mar 19, 2021 at 18:30
  • Yes, you could execute the script in the background too. Mar 19, 2021 at 20:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .