Append and write out variable-named file from list of variables

Question

I would like to create multiple 'wget' shell scripts to download a batch of considerably large (~3GB) files using our HPC cluster. The names of the files are stored in filenames.txt, like so:

$cat filenames.txt
file1
file2
file3
...

and the urls I would like to wget from are structured like so:

ftp://host.com/dir1/dir2/file1/file1.sra
ftp://host.com/dir1/dir2/file2/file2.sra
ftp://host.com/dir1/dir2/file3/file3.sra

I'd like to create a shell script for each wget and write it out into a file named as the variable itself. For example, file1.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

and file2.sh should contain:

#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

As you can see, the pattern to match is 1) The URL and 2) the filename to be written out. How would I 'append' the URL to the filename, then write it out into a .sh file named after it?

Answer 1

You can make a very simple shell loop to do this:

while read filename
do
    echo '#!/bin/bash' > $filename.sh
    echo "wget ftp://host.com/dir1/dir2/$filename/$filename.sra" >> $filename.sh
done < filenames.txt

This reads each line from filenames.txt and calls it filename, then for each one writes out a file named $filename.sh, where $filename is replaced with the line from the file. That file has two lines: #!/bin/bash, in every file, and then the wget command you wanted, again with the file name substituted in. >> is appending that second line into the same file, rather than overwriting. After running this script:

$ cat file1.sh 
#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra
$ cat file2.sh
#!/bin/bash
wget ftp://host.com/dir1/dir2/file2/file2.sra

---

You may want to add a third line inside the loop body:

chmod a+x "$filename.sh"

to make the scripts be executable afterwards. Put that immediately above the done line.

---

If any of your file names have spaces or other special characters in them then this will fall apart (on multiple levels), but for alphanumeric names it will be fine.

Answer 2

Use this command:

awk -v url='ftp://host.com/dir1/dir2' '{printf "#!/bin/bash\nwget %s/%s/%s.sra\n",url,$1,$1 >$1".sh"}' filenames.txt

After this command is run, there will be a series of files in the current directory like:

$ ls *.sh
file1.sh  file2.sh  file3.sh

The contents of each looks something like:

$ cat file1.sh
#!/bin/bash
wget ftp://host.com/dir1/dir2/file1/file1.sra

How it works

• -v url='ftp://host.com/dir1/dir2'

  This defines a url as an awk variable.

• printf "#!/bin/bash\nwget %s/%s/%s.sra\n",url,$1,$1 >$1".sh"

  This prints out each file that you need. The >$1".sh" part means that each shell script is written to a file named after the file it downloads with a .sh extension added.

Answer 3

Yet another proposition, using for loop, printf, and command substitution $():

for file in $(<filenames); do
    printf "%s\n%s\n" '#!/bin/bash' "wget ftp://host.com/dir1/dir2/${file}/${file}.sra" > "${file}.sh"
done

The only non-trival part is probably $(<filename), what is equivalent of $(cat filename), but a little bit faster.