I'm using cURL to login to several websites.

Looking through the DOM manually to get the fields for the username and password fields is tedious for my case.

I'll then parse the fields and use the curl -d command to login.

The only reason I need to login is so I can get cookies. The same is true with Python for some sites.

cURL, gets cookies in netscape format, with Python you have to do more work to convert to netscape format.

  • I have rolled back your edits (adding a different question and its answer to the question text). If you want to ask a different question and want to answer it with your own answer, then you are free to do so (see unix.stackexchange.com/help/self-answer), but do so by opening a new question.
    – Kusalananda
    Feb 26 at 4:14
  • a different mod erroneously asked me to add an answer to this question. I told the poor fellow that this question had been answered and that adding an answer for my actual issue would not make sense. I'd like to thank you for pointing me to the help link, I could never have found that myself, I am forever in your debt, many thanks. However, I did not add another question in the text, it can be difficult to comprehend the meaning of text at times so this error you have made is forgiven and no hard feelings.
    – Nickotine
    Feb 27 at 17:40

2 Answers 2


Curl doesn't do the HTML parsing for you in this case.

You need to do it yourself; this suggests maybe trying to do this all in a shell script is not the easiest route to take.

I'd probably go an write a short Python program using the requests library to do the http part, and the BeautifulSoup library to get the fields out of the DOM.

  • ok thanks, do you have any suggestions for what I wrote to the other answerer?
    – Nickotine
    Feb 25 at 1:37
  • As I already said in my answer: time to stop trying to do all the things in shell scripts, no matter how many you already have. You can write your own command line tool trivially using the libraries described in both answers.
    – sina bala
    Feb 25 at 2:30
  • the shell script thing I have is more like a main file with other scripts as modules, it is something I've been using for years. Anyway I got what I wanted. In Python you first have to get the cookies, then convert to netscape format (which is a hassle, then write to a file which is more work (all these things for my purposes. I like Python/code and use that for stuff as well.
    – Nickotine
    Feb 25 at 9:06
  • You are obviously very capable and of sound mind (this is a speculation but in good faith I'll assume it's true), I'd like to thank you for answering the question while also taking your time to add extraneous information, that is an admirable thing you done, above and beyond what was required, I am very grateful. You stated 'time to stop using shell scripts for everything,' this is an easy mistake; you don't know my history, or how I work, yet arrived at this hasty conclusion. However, this is an easy error to make, we are all human after all, so all is forgiven.
    – Nickotine
    Feb 25 at 9:25

No. curl only retrieves data. It is not made to parse the response body at all.

For any sorts of automating logins to websites in scripts, I recommend using Python and the requests module as well as lxml's HTML parser.

  • Ok sure, the problem is I have a bunch of bash scripts, and this is a new feature I want to add. are there any cmdline tools which can do this?
    – Nickotine
    Feb 25 at 1:36
  • You can try to use tools like xmlstarlet or even plain old grep to search for HTML form tags and the fields therein. I can only recommend – by personal and sometimes painful experience – do not do this in bash.
    – Hermann
    Feb 25 at 19:14
  • I've done this with grep, sed and awk before. Anyway I actually didn't need to login for my purposes of extracting cookies, just needed to pass in the user-agent and get cookies in netscape format. curl gets cookies in netscape format by default, with python you have to quite a bit of work s to convert to netscape then write to a file, compared to a simple curl call with a file to write to.
    – Nickotine
    Feb 25 at 20:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .