Here's one way to do it with perl using the LWP, JSON, and Term::ANSIColor modules. Term::ANSIColor is a core perl module, but both LWP and JSON are CPAN modules. They're very commonly used modules so are probably available pre-packaged for your distro (e.g. on debian etc, apt-get install libjson-perl libwwww-perl)
#!/usr/bin/perl
use strict;
use LWP::UserAgent;
use JSON;
use Term::ANSIColor;
my $bold = color('bold');
my $reset = color('reset');
my $base='https://uk.reuters.com'
foreach my $url (@ARGV) {
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);
if ($res->is_success) {
foreach my $h ( @{ decode_json($res->content)->{headlines} }) {
print $bold, $h->{headline}, $reset, "\n", $base, $h->{url}, "\n\n";
};
} else {
die "Error processing '$url': ", $res->status_line, "\n";
}
}
This doesn't need curl or wget or multiple invocations of awk and/or cut (that ugliness was what motivated me to write an answer - as a general rule, if you're piping grep or awk into themselves then you're doing it wrong. ditto for piping cut or grep into awk - awk can do everything that those two can do and more. As can perl), or anything else.
Save it as, e.g. ./bold-2nd.pl, make it executable with chmod, and run it like this:
$ ./bold-2nd.pl https://uk.reuters.com/assets/jsonWireNews
RBS to pay $125 million to settle California mortgage bond claims
https://uk.reuters.com/article/uk-rbs-settlement/rbs-to-pay-125-million-to-settle-california-mortgage-bond-claims-idUKKBN1EH053
Driver charged with attempted murder over Australian vehicle attack
https://uk.reuters.com/article/uk-australia-attack/driver-charged-with-attempted-murder-over-australian-vehicle-attack-idUKKBN1EH044
EasyJet says other airlines interested in feeder flights from Tegel
https://uk.reuters.com/article/uk-air-berlin-m-a-easyjet/easyjet-says-other-airlines-interested-in-feeder-flights-from-tegel-idUKKBN1EH04W
[...]
This version of the script can handle multiple URLs on the command line (of course, they all need to return the same json-formatted data...or at least extremely similar with both a headline and a url field).
btw, I've made it print a blank line between each article. I find that to be more readable.
If you want to use curl to do the fetching rather than the perl LWP module, the script would be quite a bit simpler:
#!/usr/bin/perl
use strict;
use JSON;
use Term::ANSIColor;
my $bold = color('bold');
my $reset = color('reset');
my $base='https://uk.reuters.com'
undef $/;
my $json = <>; # slurp in entire stdin
foreach my $h ( @{ decode_json($json)->{headlines} }) {
print $bold, $h->{headline}, $reset, "\n", $base, $h->{url}, "\n\n";
};
Run this version as:
$ curl -s https://uk.reuters.com/assets/jsonWireNew | ./bold-2nd.pl
Note that both versions of the bolding script use a json parser to actually parse the json data, rather than relying on regular expressions to search for lines matching particular patterns. As has been noted many times before, parsing json, or html, or xml, or any similar structured data format with regular expressions is unreliable and fragile. In the simple case, it can be made to work but even minor changes in the input format can break the script (e.g. if Reuters stops outputting pretty-printed json with line feeds between each data element and record, and starts printing just a single line of json, any line-based regexp pattern matcher will break)
Finally, the json data fetched by curl (or LWP) looks like this:
{ "headlines": [
{ "id": "UKKBN1EH044",
"headline": "Driver charged with attempted murder over Australian vehicle attack",
"dateMillis": "1514003249000",
"formattedDate": "3m ago",
"url": "/article/uk-australia-attack/driver-charged-with-attempted-murder-over-australian-vehicle-attack-idUKKBN1EH044",
"mainPicUrl": "https://s4.reutersmedia.net/resources/r/?m=02&d=20171223&t=2&i=1216634499&w=116&fh=&fw=&ll=&pl=&sq=&r=LYNXMPEDBM04W"
},
]}
so, id, dateMillis, formattedDate, and mainPicURL are also available for printing or other use in the perl $h hashref variable, as well as the headline and url that we're printing.
