4

I'm piping the output of a curl request to gawk so that I can pull some data. The gawk code is already working if I'm searching through the html file which curl request produces. However, I was hoping to simply edit in place, as I just need to allocate 2 date variables. I've been looking at using tee and process substitution. Is there any way to somehow fork the output of the curl request in order to assign 2 different variables, both using gawk and command substitution.

Here's a snippet of my code, I've hidden the full details of the curl request, but it is going to pull an html report which will contain: Oldest Sequence, date and Newest Sequence, date on separate lines. The gawk has already been tested on the data.

curl -k -q "https://user:password@ipaddress/report"

I want the output of the above command to be piped or otherwise to the two commands below

date1="$(date -d $(gawk '/Newest Sequence/ {print $3,$4}') +%s)"
date2="$(date -d $(gawk '/Oldest Sequence/ {print $3,$4}') +%s)"

Is this possible without creating a file? I will be running the request at frequent intervals and sending the result to a socket, I'd rather avoid unnecessary files if possible

Improve this question

1 Answer 1

Reset to default
4

The basic syntax you need is this:

read date1 date2 < <( curl ... | gawk '...' )

This way you need just one awk instance as illustrated here (without the seconds conversion which you'd have to add; see below):

read date1 date2 < <( curl ... |
    awk '
      /Newest Sequence/ { new=$3" "$4 }
      /Oldest Sequence/ { old=$3" "$4 }
      END { print new, old }
      '
    )

(If the order of the dates in the HTML file is fixed that could be simplified by immediately printing the date information.)

Note that gawk has also the necessary time functions so that your date based code will become obsolete. In the code I showed you'd need to add awk's mktime() calls, or alternatively (to avoid gawk's time functions) do the conversion on shell level with date modifying the variables, as in:

date1=$(date -d "${date1}" +%s)
date2=$(date -d "${date2}" +%s)
Improve this answer
16
  • I need to filter the output of of gawk twice to get 2 dates. one from oldest sequence and one from newest sequence. And yes, I should use the gawk dates, it's that horrible stfrtime function I remember from one of my first c++ assignments, when I didn't know any c at all. Need to read through properly and then could use instead
    – jewfro
    Mar 26, 2015 at 5:01
  • 2
    @Archemar; in ksh (e.g.) you can do that, because the final pipe command is performed in the current shell process (and not spawned in a subprocess), but in bash - the shell that seems to dominate here - that's not possible; because date1 and date2 will in bash not be set in the current process you can't simply (without other means) use them after the command. Try echo a b | read x y ; echo $x $y once in ksh and once in bash and you'll see the difference.
    – Janis
    Mar 26, 2015 at 6:38
  • 1
    @jewro; WRT your first question today: the direct read from pipe approach will not work in bash (because of the subshell issue I described). If you can use a ksh then I'd go for the read from pipe, just because it's a simpler construct. WRT performance don't worry; in both constructs you have a communication pipe (either explicit or implicit), and the same number of processes.
    – Janis
    Mar 27, 2015 at 5:49
  • 1
    @jewro; WRT your second question today: well, in this specific case I think it's a bit simpler to do it in shell (as illustrated in my answer; for the difference just add diff=$((new-old))), because in awk you'd have to split the date format from the HTML file to fit the syntax of awk's mktime() function, which might get a bit clumsy. Though if you intend to do yet more data processing it may pay to do everthing in awk. So it depends. It is certainly a good example to try and enlightening to check out both variants.
    – Janis
    Mar 27, 2015 at 5:57
  • 1
    In the code of your comment I don't see any read command used; check that. In the awk program the default OFS is already a blank, so the print new, old should work as expected. (The spurious semicolon must not be there, yes.) To find the bug I suggest to start with the plain syntactical form, and add your commands one by one, so that you can see where the problem is.
    – Janis
    Mar 30, 2015 at 4:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .