I've got a text file from which I'd like to extract some lines and phrases. I'd also like to insert a couple of characters in some spots. It's a file I intend to download periodically over the course of the day, one with certain values that change throughout the course of the day: the file is a simple description of weather conditions, giving things like current time, temperature, humidity and so forth. Most of the text in the file is not of interest and I'm trying to find a way to get from it just the text I'm interested in. I grab the file using curl and I'd like to pipe it through some command(s) that will extract only the needed parts and insert the characters mentioned.

The content of the text file I grab looks a lot like this:

Smyrna, Smyrna Airport, TN, United States (KMQY) 36-00-32N 086-31-12W
Aug 09, 2016 - 10:56 AM EDT / 2016.08.09 1456 UTC
Visibility: 10 mile(s):0
Sky conditions: partly cloudy
Temperature: 86 F (30 C)
Heat index: 93.4 F (34.1 C):1
Dew Point: 73 F (23 C)
Relative Humidity: 66%
Pressure (altimeter): 30.01 in. Hg (1016 hPa)
ob: KMQY 091456Z 18006 10SM SCT035 30/23 A3001
cycle: 15

Of those lines, only 2, 4, 5, and 8 contain text of interest--the other lines can just be discarded. Furthermore, a lot of what's in the target lines is not needed as well. It should be remarked that the content of this file will be somewhat fluid, with variables like time, temperature and humidity changing at regular intervals. I'd like the finished product, after uneeded material has been eliminated, to look as follows:

'Aug 09, 2016 - 10:56 AM EDT\nconditions: partly cloudy\n86 F\nHumidity: 66%'

Note the inserted characters-- \ n and ' (inverted commas at the beginning and end)--characters that need to be inserted to indicate line breaks and that the entire line, including spaces, is to be included, to the program to which I'll be feeding this output (imagemagick). Preferably, I'd like to be able to pipe the output from curl to some command or series of commands that will extract that target material, insert the inverted commas and \n sequences, then write that to a file. So, something like

curl http://my.weat.her/local.txt | command(s) > currentcond.txt

Anyone have suggestions for accomplishing this task?

1 Answer 1


If the output of the curl command is stable and consistent then we can build out a very very basic set of sed commands based on line number


2s! /.*!!

would take line 2 and delete everything after the / and so convert

Aug 09, 2016 - 10:56 AM EDT / 2016.08.09 1456 UTC


Aug 09, 2016 - 10:56 AM EDT

We can add \\n to the replacement.

Then use the -n option to sed to make it only print lines that have the p on the replacement

So we end up with:

sed -n -e '2s! /.*!\\n!p' \
       -e '4s/Sky \(.*\)/\1\\n/p' \
       -e '5s/.*: \([0-9]* F\).*/\1\\n/p' \
       -e '8s/Relative //p'

Finally we need to remove the linefeed characters with tr -d '\012' and for simplicity we'll put it all on one line:

sed -n -e '2s! /.*!\\n!p' -e '4s/Sky \(.*\)/\1\\n/p' -e '5s/.*: \([0-9]* F\).*/\1\\n/p' -e '8s/Relative //p' | tr -d '\012'

This solution is fragile if the input may vary, but it's pretty easy to understand.

So let's look at something a little less fragile and try to detect the lines based on pattern.

awk '/UTC$/ { gsub(/ \/.*$/,""); d=$0 }                      
     /^Sky conditions:/ {gsub(/^Sky /,""); s=$0 }
     /^Temperature: / { t=$2 }
     /Relative Humidity: / { h=$3 }
     END { print d"\\n"s"\\n"t" F\\nHumidity: "h }

Here the order of the lines aren't so important as long as they match the pattern; dates must end in UTC, etc etc

EDIT based on updated question.

Putting quotes inside awk can sometimes be painful, so we can cheat and just assign it to a variable before hand (q in this case) and then use that directly in the final print statement

awk -vq=\' '/UTC$/ { gsub(/ \/.*$/,""); d=$0 }                      
            /^Sky conditions:/ {gsub(/^Sky /,""); s=$0 }
            /^Temperature: / { t=$2 }
            /Relative Humidity: / { h=$3 }
            END { print q""d"\\n"s"\\n"t" F\\nHumidity: "h""q }
  • Great suggestions, Stephen Harris. I'd come up with a double pipe using sed; sed -e '1d;3d;4d;7d;9,11d' | sed 's:/.*::' to get something close. But I still had not figured out how to insert the \n. Your awk suggestion does a wonderful job on that. But I neglected one other addition I need to make to the final output: it should be surrounded by single quotes. I'll edit my original post now to reflect that.
    – MJiller
    Aug 9, 2016 at 17:57
  • Updated the awk solution with single quote wrapping Aug 9, 2016 at 18:10
  • The revised version works great--thanks for putting that together for me. Much appreciated. I've run into an unanticipated problem though, and it's got nothing to do with my idea of extracting text and doing a bit of reformatting. I think rather that it has to do with limitations of the for loop I've been using to feed content to imagemagick's convert. As long as what I feed it in my for loop is a single word or group of characters, all works fine. But it seems the loop, when confronting white space, starts breaking words/groups into separate lines. And convert can't deal with those lines
    – MJiller
    Aug 9, 2016 at 22:59
  • Sounds like the basis for another question :-) Aug 9, 2016 at 23:55
  • I got the answer to the other question in a different posting here--turns out I was trying to feed the file content to imagemagick wrongly. And it turns out that, when I feed that file content in the proper way, inverted commas are uneeded. My bash IQ is quite low, as you might guess. I've actually decided I'd like for the dash and spaces between 2016 and 10:56 to be replaced in the output by \n (date and time end up on different lines, which looks better in the final output). I'll see whether I can puzzle out how to further revise your awk statement to accomplish that.
    – MJiller
    Aug 10, 2016 at 13:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .