2

Background

I've been using lynx for a couple of years to dump automatically (via cron), process (using awk, sed, and tr), then mail (bsd-mailx) the contents of a web page. It's been working pretty well--until recently, when the site security certificate expired and the idiotic web admins didn't bother to update it. Now the automated solution is failing owing to lynx balking at the certificate and expecting user input before proceeding.

Attempted solution

So I started looking into other solutions. curl looked promising since it has an --insecure switch. In fact, using it I'm able to download text of the page and further process it into more standard text output that could go into the body of an e-mail, then save that to a file.

The problem I'm having with that, however, is that this file is somehow being seen by the system as charset=binary (according to output of file -i) --possibly because it contains text in a non-Latin alphabet (cyrillic). Therefore, when I try to cat the file content into the body of an e-mail I send to myself, the mail program (bsd-mailx) wants to treat it as an attachment -- not what I want -- rather than inserting it into the body of the e-mail.

Since I'm having difficulty grasping all the issues involved, I'm not finding a solution so far.

Questions

  • Is there some way to cause the file to not be saved/identified as binary?
  • Or to somehow convert it from binary to some valid charset?

    • Note that if I copy the content of the file to the clipboard, then paste it into a new text file I've created it all works: file -i shows charset=utf-8 and I can successfully cat the content into an e-mail's body without it being treated as an attachment. But I'm looking for something I can automate as a cron job, so that isn't really the solution I'm hoping for.

PS: I recall having had a similar problem under lynx, but that was resolved, IIRC, by adding the switch -display_charset=UTF-8 when invoking lynx.

Update

Here, using Lucas' suggestion, is what works:
curl --insecure https://my.url.html >/home/user/file.html lynx -display_charset=UTF-8 -dump -nonumbers -nolist /home/user/file.html | sed fu begin | sed fu end | tr fu >/home/user/file.txt cat /home/user/file.txt | mail -s "Today's file" me@myemail.addy
A pretty klunky two-stage process and I'm sure there must be easy ways to streamline and make this more elegant, but it does get the job done. Can anyone offer improvements? Obviously sed fu and tr fu are stand-ins for the actual sed and tr commands I'm using, which are mainly stripping out html tags and extraneous whitespace.

  • Got lost, is there a way of having less context and getting to the point? – Rui F Ribeiro Jul 23 '18 at 19:21
  • 1
    Try to download to a tempfile with curl, the open that tempfile with lynx and continue like before. If that is what you do might need to see the minimal version of you script that displays the problem. – Lucas Jul 23 '18 at 19:25
  • 2
    That's a pretty brilliant and quite simple suggestion, Lucas: some testing reveals that it works (can save it as charset=UTF-8)! But it does make the already somewhat convoluted process a wee bit more unwieldy. If possible I'd like to simplify things, but it's good to know that there's something fairly simple that will work. Sorry to have taxed your reading comprehension with those 334 words, Rui F. Ribeiro. I'll see if I can simplify it a bit further – MJiller Jul 23 '18 at 19:34
  • I'm using the solution described under the "Update" section to good effect so far. I would mark my own question as the solution if that were possible. Since it's not, I'm appending this comment as an indicator for the solution I found. – MJiller Jul 31 '18 at 0:02
2

Use HTTPie instead of curl and lynx. This HTTP client has --verify no option to skip SSL certificate verification. It's very simple to use HTTPie:

http --verify no -F https://google.com

Additionally see HTTPie web-site and some nice examples of using. More information on HTTPie options:

man http

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.