Keep global variables values piping through functions

Question

I wrote a little bash script using sed on some html pages to extract some urls.

To avoid each time grabbing sed results in a variable then read it again I simply made 3 functions and piped together.

first_function $1 | second_function | third_function

Let's say that:

• the first function finds item urls in a list from a given href ($1)
• the second function extracts from each of the piped urls an image src
• the third function assembles output HTML

at the moment I echo $lot_url in second_function so I can read it in third_function along with $img_url and put it in HTML.

It would be cleaner if I could just hold it as a global variable that's accessible from second_function to third_function but seems I can not.

This becomes more necessary as the number of values to pass between functions grows.

Here a full sample code:

first_function(){
    curl -s "$1" | sed -nr '
        #extract sub urls
    '
}

second_function(){
    while read lot_url; do
        echo "$lot_url"
        curl -s "$lot_url" | sed -nr '
            #extract img src
        '
    done
}

third_function(){
    while read lot_url; read img_url; do
        echo "<a href="$lot_url"><img src="$img_url" /></a>"
    done
}

first_function "$1" | second_function | third_function

Answer 1

The two sides of a pipe are in different processes. You can't share variables between these processes. If you want to share data, you either have to pass it through the pipe, or use alternate communication channels. If you need alternate communication channels, you're above the shell's capabilities, switch to a real programming language.

Here, passing lot_url alongside img_url in the second pipe seems like a good solution to me. I'd pass them on the same line. Assuming your URLs are properly escaped, you don't need any particular quoting, you can pass them on the same line. This would have the advantage of allowing a variable number of img_urls in each lot_url.

second_function(){
    while read lot_url; do
        echo "$lot_url"
        curl -s "$lot_url" | sed -nr -e '
            #extract img src
        ' -e "s>^>$lot_url >"
    done
}

third_function(){
    while read lot_url img_url; do
        echo "<a href="$lot_url"><img src="$img_url" /></a>"
    done
}

Answer 2

A single while loop would do here, I think. I think it would make no difference, really, since you're calling executables every iteration anyway. In this way you can share the global variable you had hoped for. Like:

source_cmd |
    while read var
    do  fn1 "$var" |
        fn2 "$var"
    done

But I think better still would be to change the workflow a little - the function is nothing more than a set of shell commands based around an array. Now you're not using the array for any of them, so it should really serve to point out a common purpose for a set of commands - and the most common that I can see seems to be curl ... | sed ... . So I suggest you should make that a function that can accept parameters. So the result might look like this:

curl_sed() { url=$1 && shift
    curl -s "$url" | sed -nr "$*"
}
fn() { URL=$1 && shift
    set -- '#extract sub url sed script' \
           '#extract img src sed script'
    curl_sed "$URL" "$1" | 
        while read lot_url
        do  IFS='
';          printf '<a href="'"$lot_url"'"><img src="%s" /></a>\n' \
            $(curl_sed "$lot_url" "$2")
        done
}

You've already accepted Gilles's answer to this question - apparently years ago, which I didn't realize - but here's another that demonstrates similar methodology to mine, and which I think would be better suited to this purpose.