1

I used this script for downloading files. Without -b, wget download files one by one. With -b, I have the possibility to download files in background but also simultaneously. Unfortunately, the script doesn't work in SLURM. It only works without -b in Slurm.

Script for downloading files

#!/bin/bash

mkdir data
cd data
for i in 11 08 15 26 ;
do
  wget -c -b -q ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR116/0${i}/SRR116802${i}/SRR116802${i}_1.fastq.gz 
  wget -c -b -q ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR116/0${i}/SRR116802${i}/SRR116802${i}_2.fastq.gz
done

cd ..

Slurm Script

#!/bin/bash

#SBATCH --job-name=mytestjob     # create a short name for your job
#SBATCH --nodes=2                # node count
#SBATCH --ntasks=2               # total number of tasks across all nodes
#SBATCH --cpus-per-task=2        # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem-per-cpu=4G         # memory per cpu-core (4G is default
#SBATCH --time=10:01:00          # total run time limit (HH:MM:SS)
#SBATCH --array=1-2              # job array with index values 1, 2

#Execution

bash download.sh

On the terminal : sbatch slurmsript.sh ( It dosen't work) no jobid

1 Answer 1

1

using job array you can download files simultaneously (e.g.one download per array task). No need to put wget to the background. In my experience, actually, if you try to 'release' the slurm script command line by launching a process in the background, slurm will terminate the job when the slurm submission script finishes execution ( but your tasks may still be running in the background). So it is foreground you want with slurm!

1
  • Thank you for this for this well-argued explanation.
    – S.AZOUZ
    Feb 25, 2021 at 3:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .