1

Hi there I'm trying to download a large number of files at once; 279 to be precise. These are large BAM (~90GB) each. The cluster where I'm working has several nodes and fortunately I can allocate multiple instances at once.

Given this situation, I would like to know whether I can use wget from a batch file (see example below) to assign each download to a separate node to carry out independently.

batch_file.txt

<https_link_1> -O DNK07.bam
<https_link_2> -O mixe0007.bam
<https_link_3> -O IHW9118.bam
.
.

In principle, this will not only speed up things but also prevent the run from failing since the wall-time for this execution is 24h, and it won't be enough to download all those files on a single machine consecutively.

This is what my BASH script looks like:

#!/bin/bash
#
#SBATCH --nodes=279 --ntasks=1 --cpus-per-task=1
#SBATCH --time=24:00:00
#SBATCH --mem=10gb
#
#SBATCH --job-name=download
#SBATCH --output=sgdp.out
##SBATCH --array=[1-279]%279
#
#SBATCH --partition=<partition_name>
#SBATCH --qos=<qos_type>
#
#SBATCH --account=<user_account>

#NAMES=$1
#d=$(sed -n "$SLURM_ARRAY_TASK_ID"p $NAMES)

wget -i sgdp-download-list.txt

As you can see I was thinking to use an array job (not sure whether will work); alternatively, I thought about allocating 279 nodes hoping SLURM would haven been clever enough to send each download to a separate node (not sure about it...). If you are aware of a way to do so efficiently, any suggestion is welcome. Thanks in advance!

0

3 Answers 3

0

Expand the command into multiple wget commands so you can send them to SLURM as a list:

while IFS= read -r url; do 
  printf 'wget "%s"\n' "$url"
done < sgdp-download-list.txt > wget.sh

Or, if your sgdp-download-list.txt is just a list of wget command missing the wget at the beginning (which is what your example suggests), just use:

sed 's/^/wget /' sgdp-download-list.txt > wget.sh

Then, submit the wget.sh as the job.

0

Frame Challenge

You are talking 25 Terabytes of download. I think you will find fetching nodes to be the least of your worries. If they are coming from the same remote site, then the remote site could also have CPU issues. More importantly, network bandwidth is an issue. If you are trying to pull this download through one pipe, it really won't matter how many nodes you have pulling - the pipe will be the limit.

Having said that, wget isn't particularly CPU intensive. Running 10 in parallel on one node shouldn't be an issue, in terms of CPU.

If this is a regular activity, I would recommend seeing if you can download deltas.

Alternative

There is an old joke from Usenet. Someone asked what the best way to daily send a large quantity of data from LA to San Fransisco. The answer: UPS. Network speeds have increased, but so has your volume. Amazon takes this to heart: The can ship you a box to load data into to, then ship back to get it in their cloud.

1
  • Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
    – doneal24
    Oct 23, 2023 at 12:59
0

Hi both @terdon and @David G.,

sorry for the late reply. In the end the tech support on the cluster recommended me an ad-hoc approach using data mover nodes with which I could download continuously till the links expire.

Then, I simply had to regenerate them and start the process again. It wasn't the fastest process overall (took about a week) but considering the huge amount of data downloaded wasn't too bad either!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .