I have the URL (see below) of a certain web page that lists many different versions of a software package.


The following one-liner gets me the latest version tar ball and its signature file out of the HTML.

wget -qO- http://ftp.gnu.org/gnu/wget/ | grep tar | cut -d\" -f6 | tail -n4 | grep gz

Probably not the shortest most efficient one liner, but hey, I'm learning and I'm open for feedback. The result of the above is this:


Now, the next logical step (to me at least), is to pipe the output above into sed and append the $URL to the front of each line so that the output looks like:


And then I want to pipe that right back into wget to download the files.

The question is this: How do I append the value of the bash variable $URL to the front of each line of output using sed? I tried the following:

sed "s/^/$URL/"

But that only gives me the error:

sed: -e expression #1, char 11: unknown option to `s'

I also know that the basic concept is good, because when I use the following, I get good results...

VAR="Gorauskas, "
echo "Jonas" | sed "s/^/$VAR/"

So, my guess is that I need to somehow escape all of the / character in the $URL variable... Am I on the right track?

5 Answers 5


Use a different separator that doesn't contain any of the characters in the variable.

For example,

sed "s|^|$URL|"

(If you use / as the separator and the pattern or replacement also contain /, then you'd need to escape those.)

  • It works! So easy, and yet so hard... :) Commented Feb 24, 2014 at 7:33
  • 6
    Another character to watch in URLs (in the general case, not here) is & as it's special to sed on the right hand side of a substitution. Commented Feb 24, 2014 at 8:52

What you do until now can be replaced by one awk call:

wget ... | awk -F\" '$6 ~ "gz$" { lastline=thisline; thisline=$6;}; '\
  'END {print lastline; print thisline;}'

And, of course, awk can add the URL, too:

awk -F\" -v baseurl="http://ftp.gnu.org/gnu/wget/" \
  '$6 ~ "gz" { lastline=thisline; thisline=$6;}; '\
  'END {print baseurl lastline; print baseurl thisline;}'

You can use wget's --base option here:

wget -qO- http://ftp.gnu.org/gnu/wget/ |
  cut -d\" -sf6 |
  grep '\.tar\.gz' |
  tail -n2 |
  wget -i - --base=http://ftp.gnu.org/gnu/wget/

You can also do the whole thing directly like so:

wget -qO- http://ftp.gnu.org/gnu/wget/ | grep tar.gz | cut -d\" -f6 | 
 tail -n2 | xargs -I{} wget http://ftp.gnu.org/gnu/wget/{}

This passes the output of the 1st wget to xargs which replaces the string {} with each of the results of the piped command.

And you can skip some parsing steps with some trickery:

wget -qO- http://ftp.gnu.org/gnu/wget/ | tac | grep -Pom 2 'href="\K(.+?.tar.gz)' | 
xargs -I{} wget http://ftp.gnu.org/gnu/wget/{}

Here, we are using PCREs (-P) with grep and -o so it only prints the matched portion of the line and -m 2 to only print the 2 first matches. The tac call reverses the input so that the first 2 matches are actually the last (tac reverses its input, prints the last line as first, penultimate as second etc.).

The \K in the regular expression tells grep to ignore whatever came before the \K so that it is not printed when using -o.

Another approach, closer to what you had in mind, would be to read the target files in a loop:

wget -qO- http://ftp.gnu.org/gnu/wget/ |tac | 
 grep -Pom 2 'href="\K(.+?.tar.gz)' | 
    while read target; do 
        wget http://ftp.gnu.org/gnu/wget/"$target"; 
ls | sort -n  | sed "s|^|https://url.com/dl/p/|" > dl.txt


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .