Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have the URL (see below) of a certain web page that lists many different versions of a software package.

URL=http://ftp.gnu.org/gnu/wget/

The following one-liner gets me the latest version tar ball and its signature file out of the HTML.

wget -qO- http://ftp.gnu.org/gnu/wget/ | grep tar | cut -d\" -f6 | tail -n4 | grep gz

Probably not the shortest most efficient one liner, but hey, I'm learning and I'm open for feedback. The result of the above is this:

wget-1.15.tar.gz
wget-1.15.tar.gz.sig

Now, the next logical step (to me at least), is to pipe the output above into sed and append the $URL to the front of each line so that the output looks like:

http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz
http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz.sig

And then I want to pipe that right back into wget to download the files.

The question is this: How do I append the value of the bash variable $URL to the front of each line of output using sed? I tried the following:

sed "s/^/$URL/"

But that only gives me the error:

sed: -e expression #1, char 11: unknown option to `s'

I also know that the basic concept is good, because when I use the following, I get good results...

VAR="Gorauskas, "
echo "Jonas" | sed "s/^/$VAR/"

So, my guess is that I need to somehow escape all of the / character in the $URL variable... Am I on the right track?

share|improve this question

4 Answers 4

up vote 11 down vote accepted

Use a different separator that doesn't contain any of the characters in the variable.

For example,

sed "s|^|$URL|"

(If you use / as the separator and the pattern or replacement also contain /, then you'd need to escape those.)

share|improve this answer
    
It works! So easy, and yet so hard... :) –  Jonas Gorauskas Feb 24 at 7:33
4  
Another character to watch in URLs (in the general case, not here) is & as it's special to sed on the right hand side of a substitution. –  Stéphane Chazelas Feb 24 at 8:52

What you do until now can be replaced by one awk call:

wget ... | awk -F\" '$6 ~ "gz$" { lastline=thisline; thisline=$6;}; '\
  'END {print lastline; print thisline;}'

And, of course, awk can add the URL, too:

awk -F\" -v baseurl="http://ftp.gnu.org/gnu/wget/" \
  '$6 ~ "gz" { lastline=thisline; thisline=$6;}; '\
  'END {print baseurl lastline; print baseurl thisline;}'
share|improve this answer
    
Very, very cool indeed! –  Jonas Gorauskas Feb 24 at 7:50

You can use wget's --base option here:

wget -qO- http://ftp.gnu.org/gnu/wget/ |
  cut -d\" -sf6 |
  grep '\.tar\.gz' |
  tail -n2 |
  wget -i - --base=http://ftp.gnu.org/gnu/wget/
share|improve this answer

You can also do the whole thing directly like so:

wget -qO- http://ftp.gnu.org/gnu/wget/ | grep tar.gz | cut -d\" -f6 | 
 tail -n2 | xargs -I{} wget http://ftp.gnu.org/gnu/wget/{}

This passes the output of the 1st wget to xargs which replaces the string {} with each of the results of the piped command.

And you can skip some parsing steps with some trickery:

wget -qO- http://ftp.gnu.org/gnu/wget/ | tac | grep -Pom 2 'href="\K(.+?.tar.gz)' | 
xargs -I{} wget http://ftp.gnu.org/gnu/wget/{}

Here, we are using PCREs (-P) with grep and -o so it only prints the matched portion of the line and -m 2 to only print the 2 first matches. The tac call reverses the input so that the first 2 matches are actually the last (tac reverses its input, prints the last line as first, penultimate as second etc.).

The \K in the regular expression tells grep to ignore whatever came before the \K so that it is not printed when using -o.


Another approach, closer to what you had in mind, would be to read the target files in a loop:

wget -qO- http://ftp.gnu.org/gnu/wget/ |tac | 
 grep -Pom 2 'href="\K(.+?.tar.gz)' | 
    while read target; do 
        wget http://ftp.gnu.org/gnu/wget/"$target"; 
    done 
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.