I can get a url link and search for text that starts with file: but I'm having issues parsing it from there.


wget -qO- http://website.com/site/ | tr \" \\n | grep -w file:\* > output.txt

The wget command Gives me the output:

file: 'http://website.com/site/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyXs',

I'm trying to get the output to look like.


My goal is to have a bash script that includes several source / list of url's that will be looped through and each processed / grep'd output url will be on it's own separate line.


As requested:

Here's an example of the output of what wget -qO- http://website.com/site/ sends back.

  file: 'http://website.com/site/myStream/playlist.m3u8?wmsAuthSign=c2VydmVyXs',
  width: "100%",
  aspectratio: "16:9",

  • If you want to parse html, I recommend to use something made for parsing html instead of grep. Depending on the actual HTML files you want to parse, you may get away with grep, but there'll be plenty of variants your regular expression won't catch. – dirkt Aug 25 '19 at 7:20
  • 2
    can you show an example of the ACTUAL output of the wget command before any processing with tr or grep? – cas Aug 25 '19 at 7:22
  • @cas ok I updated the question to include just what the wget -qO- http://website.com/site/ with no processing outputs. – Rick T Aug 25 '19 at 7:58
  • ok, so it's not returning HTML. looks like it's returning a function call with embedded json. lynx -dump won't work for that at all. – cas Aug 25 '19 at 8:01

This will do what you want:

wget -qO- http://website.com/site/ | \
  sed -n -e "/^ *file: */ { s/^ *file: *'//; s/', *$//p}" > output.txt
  • I installed it unfortunately it creates a blank file... the line wget -qO- http://website.com/site/ | tr \" \\n | grep -w file:\* > output.txt works it just doesn't fully parse it the way I need it. – Rick T Aug 25 '19 at 7:33
  • then please add a sample of the raw output of wget to your question. Without that, only rough guesses are possible. – cas Aug 25 '19 at 7:35
  • It's highlighted in the question under Example? is it not showing up in the question? – Rick T Aug 25 '19 at 7:36
  • also, does the lynx command about without the grep show a list of links? does your http://website.com/site/ require authentication? if so, you might need to use the -auth=ID:PASSWD option. or visit the site manually with lynx using the --cookie_file option and then use the same cookie file with the lynx -dump ... later. – cas Aug 25 '19 at 7:38
  • 1
    yeah, it works at the moment. It's fragile, though - any change in the data returned by the web site could and probably will break it. e.g. just removing all linefeeds from the data will a) probably be valid for the official/expected client software, but b) break the above until you edit the sed script to cope with the new situation. – cas Aug 25 '19 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.