I am trying to get html data from servicenow website. It's using frames and I would like to extract the frame source using wget. I am unable to find the option/flags to do this job. I am only able to extract the page source but not the frame source.

wget -q https://company.service-now.com/task.do?sysparm_query=number=TASK0299719

Chrome browser snippet of right-click on website


A frame is a separate page with its own URL, just like a regular link. By using a frame, the creator of the first page is saying that the second page should be retrieved immediately and displayed within the first page, rather than waiting for the user to click on it and displaying it on its own.

Since wget doesn't display anything or have an interactive user, the distinction between frame and link is meaningless. Therefore you can think of this problem as a special case of "I want wget to follow a link".

wget has a recursive mode (-r) in which it follows links from the original URL and downloads them too. You probably don't want to download all of the linked URLs, so you need to narrow it down.

If this is a one-time operation, or if the frame's URL is stable, you can simply read the original page source yourself, find the iframe element's src attribute, and ask wget to get that.

If you need wget to do more of the work for you, can use the --follow-tags option to ask it to only get iframe URLs.

wget -r --follow-tags=iframe http://yourpage/

This will download the page and all of the iframes contained in it. If there are iframes inside the iframes, it will get those too, unless you also specify a recursion depth limit like -l 1.

  • It doesn't work. I tested your solution and it still give the same output as my original command. Infact I noticed something else too. The number of lines in the view page source on browser is around 3000. But when i use wget in linux, no matter what the output i get is 361 lines. A major chunk of html is missing in wget method. I'm clueless. – ayrton_senna Nov 1 '18 at 17:55
  • And what does the HTML say? Maybe you need to authenticate and the webserver is only sending back a 401 Response? – darnir Nov 7 '18 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.