3

I would like to create multiple 'wget' shell scripts to download a batch of considerably large (~3GB) files using our HPC cluster. The names of the files are stored in filenames.txt, like so:

$cat filenames.txt
file1
file2
file3
...

and the urls I would like to wget from are structured like so:

ftp://host.com/dir1/dir2/file1/file1.sra
ftp://host.com/dir1/dir2/file2/file2.sra
ftp://host.com/dir1/dir2/file3/file3.sra

I'd like to create a shell script for each wget and write it out into a file named as the variable itself. For example, file1.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

and file2.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

As you can see, the pattern to match is 1) The URL and 2) the filename to be written out. How would I 'append' the URL to the filename, then write it out into a .sh file named after it?

3 Answers 3

0

You can make a very simple shell loop to do this:

while read filename
do
    echo '#!/bin/bash' > $filename.sh
    echo "wget ftp://host.com/dir1/dir2/$filename/$filename.sra" >> $filename.sh
done < filenames.txt

This reads each line from filenames.txt and calls it filename, then for each one writes out a file named $filename.sh, where $filename is replaced with the line from the file. That file has two lines: #!/bin/bash, in every file, and then the wget command you wanted, again with the file name substituted in. >> is appending that second line into the same file, rather than overwriting. After running this script:

$ cat file1.sh 
#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra
$ cat file2.sh
#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

You may want to add a third line inside the loop body:

chmod a+x "$filename.sh"

to make the scripts be executable afterwards. Put that immediately above the done line.


If any of your file names have spaces or other special characters in them then this will fall apart (on multiple levels), but for alphanumeric names it will be fine.

0
0

Use this command:

awk -v url='ftp://host.com/dir1/dir2' '{printf "#!/bin/bash\nwget %s/%s/%s.sra\n",url,$1,$1 >$1".sh"}' filenames.txt

After this command is run, there will be a series of files in the current directory like:

$ ls *.sh
file1.sh  file2.sh  file3.sh

The contents of each looks something like:

$ cat file1.sh
#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

How it works

  • -v url='ftp://host.com/dir1/dir2'

    This defines a url as an awk variable.

  • printf "#!/bin/bash\nwget %s/%s/%s.sra\n",url,$1,$1 >$1".sh"

    This prints out each file that you need. The >$1".sh" part means that each shell script is written to a file named after the file it downloads with a .sh extension added.

1
  • just thought I'd let you know I tried this and it worked too. And since I'm very new to awk, the explanations were helpful. Thanks.
    – aish1249
    Feb 3, 2015 at 16:11
0

Yet another proposition, using for loop, printf, and command substitution $():

for file in $(<filenames); do
    printf "%s\n%s\n" '#!/bin/bash' "wget ftp://host.com/dir1/dir2/${file}/${file}.sra" > "${file}.sh"
done

The only non-trival part is probably $(<filename), what is equivalent of $(cat filename), but a little bit faster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.