7

We can open a website with GET parameters in the browser as follows

#!/bin/bash

echo 'enter username'
read username

firefox "https://github.com/${username}"

This comes in handy because I can now visit any user's github page with just a command and then entering their username. Similarly we can make a shell script to search Google with our passed query in the parameters.

How do I open a website which requires POST parameters to be passed so that I can directly visit the website from the terminal?

Take for example, https://www.startpage.com . If POST request passing is possible then we can directly search our query from terminal.

Note: Not looking for answers based on curl to retrieve data, but answers based on firefox or any other browser to visit the website


Any other way better than Selenium because user would not have control over low level data being passed in POST request like User-Agent, lang, and some other header parameters. User would be bound to only UI options if using Selenium and these low level header can't be modified according to need.


xdotool would be costly because user would have to count how many times to do Tab in order to reach the particular form field, and then loop Tab that many times before typing something in there. It also doesn't give me the ability to change low level POST parameters like User-Agent , lang , etc

4 Answers 4

7
+100

You create a temporary auto-submitting HTML page, point the browser to that page, and after a couple of seconds, you remove the temporary HTML file as it is no longer needed. In script form:

#!/bin/bash

# Create an autodeleted temporary directory.
Work="$(mktemp -d)" || exit 1
trap "cd / ; rm -rf '$Work'" EXIT

# Create a HTML page with the POST data fields,
# and have it auto-submit when the page loads.
cat > "$Work/load.html" <<END
<!DOCTYPE html>
<html>
 <head>
  <title>&hellip;</title>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <script type="text/javascript">
   function dosubmit() { document.forms[0].submit(); }
  </script>
 </head>
 <body onload="dosubmit();">
  <form action="https://www.startpage.com/do/asearch" method="POST" accept-charset="utf-8">
   <input type="hidden" name="cat" value="web">
   <input type="hidden" name="cmd" value="process_search">
   <input type="hidden" name="language" value="english">
   <input type="hidden" name="engine0" value="v1all">
   <input type="hidden" name="query" value="&#34;Nominal Animal&#34;">
  </form>
 </body>
</html>
END

# Load the generated file in the browser.
firefox "file://$Work/load.html"

# Firefox returns immediately, so we want to give it a couple
# of seconds to actually read the page we generated,
# before we exit (and our page vanishes).
sleep 2

Let's change the above, so that we do a StartPage search on whatever string(s) are supplied on the command line:

#!/bin/bash

# Create an autodeleted temporary directory.
Work="$(mktemp -d)" || exit 1
trap "cd / ; rm -rf '$Work'" EXIT

# Convert all command-line attributes to a single query,
# escaping the important characters.
rawAmp='&'   ; escAmp='&amp;'
rawLt='<'    ; escLt='&lt;'
rawGt='>'    ; escGt='&gt;'
rawQuote='"' ; escQuote='&#34;'
QUERY="$*"
QUERY="${QUERY//$rawAmp/$escAmp}"
QUERY="${QUERY//$rawQuote/$escQuote}"
QUERY="${QUERY//$rawLt/$escLt}"
QUERY="${QUERY//$rawGt/$escGt}"

# Create a HTML page with the POST data fields,
# and have it auto-submit when the page loads.
cat > "$Work/load.html" <<END
<!DOCTYPE html>
<html>
 <head>
  <title>&hellip;</title>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <script type="text/javascript">
   function dosubmit() { document.forms[0].submit(); }
  </script>
 </head>
 <body onload="dosubmit();">
  <form action="https://www.startpage.com/do/asearch" method="POST" accept-charset="utf-8">
   <input type="hidden" name="cat" value="web">
   <input type="hidden" name="cmd" value="process_search">
   <input type="hidden" name="language" value="english">
   <input type="hidden" name="engine0" value="v1all">
   <input type="hidden" name="query" value="$QUERY">
  </form>
 </body>
</html>
END

# Load the generated file in the browser.
firefox "file://$Work/load.html"

# Firefox returns immediately, so we want to give it a couple
# of seconds to actually read the page we generated,
# before we exit (and our page vanishes).
sleep 2

All that changes is the chunk where we use Bash string operations to replace each & with &amp;, each " with &#34;, each < with &lt;, and each > with &gt;, so that the query string can be safely written as the value attribute of the hidden input named query. (Those four suffice. It is also important to do the ampersand first, because the subsequent replacements contain ampersands. Since we emit this as the value of a hidden input, the query string is not url-encoded; it's just normal HTML content, but without doublequotes (because the value itself is in doublequotes).)

The downside of auto-submitting POST requests is that you may need to update your auto-submitting HTML page every now and then, simply because the site can change the POST variable naming and internal URLs whenever they want.

3

Based on included Firefox Marionette automation driver. Install the official python bindings:

pip2 install --user marionette_driver
#!/usr/bin/python2

from marionette_driver.marionette import Marionette
from marionette_driver import By

client = Marionette('localhost', port=2828)
client.start_session()
client.navigate("https://www.startpage.com/")

query = client.find_element(By.ID, 'query')
query.send_keys("Search Me")
submit = client.find_element(By.ID, 'submit1')

submit.click()

This scipt requires firefox is already running. Maybe you have to enable Marionette using the option --marionette.

0

lynx doesn't support POST from command line

I see only these solutions :

  1. use some web automation software, like selenium

  2. choose your language and code your simple automation :

    launch the browser passing the URL on the cmdline

wait a few second and simulate key press to fill the form fields

see for example http://www.semicomplete.com/projects/xdotool/

0
0

The Zed Attack Proxy may work for you (it's a flexible intercepting proxy, the name is unfortunate). It gets attention on SO: https://stackoverflow.com/questions/tagged/zap

This naturally leads to the problem of specifying a proxy server from the command line. Here's firefox: https://stackoverflow.com/questions/843340/firefox-proxy-settings-via-command-line#843366 - Chrome will take --proxy-server=ip:port

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .