0

If you want to get straight to the question, please skip all the way down to the bolded and italicized question past the code.

Intro:

I am experiencing problems with wget and html files. I am using Ubuntu 15.10.

I am trying to gather data from a database, but instead of using "get" search, it uses "post". This simply means that I don't have URLs to work with for wget.

I have created html file to search through "post" method to bypass the "get" restriction of the site. This means that it loads the initial html file, and since it is searching using "post", the stuff I want wget to download comes takes a while to load.

I understand that wget works by searching the html file for http links, but is there a way to get wget to let the html page load for a second or two, and continue with downloading the page.

Code:

Here is the html code that acts as my link to the page I want:

<form name="PrimarySearch" id="PrimarySearch" action="http://www.dec.ny.gov/cfmx/extapps/derexternal/spills/details.cfm?pageid=2" method="post" onsubmit="return _CF_checkPrimarySearch(this)">
    <p>
        There are two ways to search the database: by spill number or by other search criteria.&nbsp;&nbsp;
        Entering a spill number will provide the full record for one spill.&nbsp;&nbsp;
        Entering other search criteria will provide a list of spills that meet the criteria,&nbsp;
        with partial information about each.
    </p>
    <div>
        <label for="spill_num" id="spill_num">
            <strong>Search method #1:</strong> Enter a spill number:
        </label>
        <script type="text/javascript">
            <!--
            document.write('<input type="hidden" name="bjavascriptEnabled" value="1">');
            -->         
        </script>
        <input name="spill_num" id="spill_num"  type="text" value="1409515"/>
            <input type="hidden" name="pageID" value="2">
        <input type="Submit" value="Submit">
    </div>
</form>
<script>document.getElementById('PrimarySearch').submit();</script>

So if I run this html file with wget and -i, it looks for a link within the html code, without even letting the html file run.

Question:

Can I let wget use my html file as an url and load it to download it?

A prompt response would be much appreciated as I need this data ASAP.


Thanks!

Howasaur

  • wget doesn't show the pages as a browser would - it only fetches them. Offhand, I don't recall any analogous (curl, etc), which do this, either. – Thomas Dickey Dec 27 '15 at 14:34
  • Just to clarify, you can't use --post-file or --post-data? gnu.org/software/wget/manual/html_node/HTTP-Options.html – Jeff Schaller Dec 27 '15 at 15:09
  • @JeffSchaller That is actually pretty helpful, didn't know that wget had --post-file and --post-data functions. This is basically what I needed, will try it out now – howasaur Dec 27 '15 at 15:15
  • @JeffSchaller If figured out how to use --post-file and --post-data, but the output files do not have anything on them, does that mean that something is quite right? – howasaur Dec 27 '15 at 15:54
1

Big thanks to @JeffSchaller , who mentioned --post-file and --post-data, which solved the whole issue.

I was able to use either with the generic "post" method only page (http://www.dec.ny.gov/cfmx/extapps/derexternal/spills/details.cfm?pageid=2) to get wget to input the value that I wanted to access that I could not through "get" method that the site search blocks. I ditched the html code that I used earlier as wget doesn't process html files.

Code:

Disclaimer: I have not tried wget without --no-parent, --convert-links, and --html-extension, but one, some, or all helped me wget the page as a local html file

wget --no-parent --convert-links --html-extension --post-data 'spill_num=1409515' http://www.dec.ny.gov/cfmx/extapps/derexternal/spills/details.cfm?pageid=2

This downloads the page as a local html file. Solving the issue where I would get the page, but with not formatting, rendering the file as blank.

Howasaur

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.