Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I wrote a little bash script using sed on some html pages to extract some urls.

To avoid each time grabbing sed results in a variable then read it again I simply made 3 functions and piped together.

first_function $1 | second_function | third_function

Let's say that:

  • the first function finds item urls in a list from a given href ($1)
  • the second function extracts from each of the piped urls an image src
  • the third function assembles output HTML

at the moment I echo $lot_url in second_function so I can read it in third_function along with $img_url and put it in HTML.

It would be cleaner if I could just hold it as a global variable that's accessible from second_function to third_function but seems I can not.

This becomes more necessary as the number of values to pass between functions grows.

Here a full sample code:

first_function(){
    curl -s "$1" | sed -nr '
        #extract sub urls
    '
}

second_function(){
    while read lot_url; do
        echo "$lot_url"
        curl -s "$lot_url" | sed -nr '
            #extract img src
        '
    done
}

third_function(){
    while read lot_url; read img_url; do
        echo "<a href="$lot_url"><img src="$img_url" /></a>"
    done
}

first_function "$1" | second_function | third_function
share|improve this question

2 Answers 2

up vote 1 down vote accepted

The two sides of a pipe are in different processes. You can't share variables between these processes. If you want to share data, you either have to pass it through the pipe, or use alternate communication channels. If you need alternate communication channels, you're above the shell's capabilities, switch to a real programming language.

Here, passing lot_url alongside img_url in the second pipe seems like a good solution to me. I'd pass them on the same line. Assuming your URLs are properly escaped, you don't need any particular quoting, you can pass them on the same line. This would have the advantage of allowing a variable number of img_urls in each lot_url.

second_function(){
    while read lot_url; do
        echo "$lot_url"
        curl -s "$lot_url" | sed -nr -e '
            #extract img src
        ' -e "s>^>$lot_url >"
    done
}

third_function(){
    while read lot_url img_url; do
        echo "<a href="$lot_url"><img src="$img_url" /></a>"
    done
}
share|improve this answer

A single while loop would do here, I think. I think it would make no difference, really, since you're calling executables every iteration anyway. In this way you can share the global variable you had hoped for. Like:

source_cmd |
    while read var
    do  fn1 "$var" |
        fn2 "$var"
    done

But I think better still would be to change the workflow a little - the function is nothing more than a set of shell commands based around an array. Now you're not using the array for any of them, so it should really serve to point out a common purpose for a set of commands - and the most common that I can see seems to be curl ... | sed ... . So I suggest you should make that a function that can accept parameters. So the result might look like this:

curl_sed() { url=$1 && shift
    curl -s "$url" | sed -nr "$*"
}
fn() { URL=$1 && shift
    set -- '#extract sub url sed script' \
           '#extract img src sed script'
    curl_sed "$URL" "$1" | 
        while read lot_url
        do  IFS='
';          printf '<a href="'"$lot_url"'"><img src="%s" /></a>\n' \
            $(curl_sed "$lot_url" "$2")
        done
}

You've already accepted Gilles's answer to this question - apparently years ago, which I didn't realize - but here's another that demonstrates similar methodology to mine, and which I think would be better suited to this purpose.

share|improve this answer
    
hard to understand more than 2 years later... thanks for sharing anyway –  neurino Jul 22 at 20:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.