12

I've written a Norwegian markdown document:

$ file brukerveiledning.md
brukerveiledning.md: UTF-8 Unicode text

I've converted it to HTML using the markdown command:

$ markdown > brukerveiledning.html <  brukerveiledning.md 
$ file brukerveiledning.html 
brukerveiledning.html: UTF-8 Unicode text

However, Firefox insists on using the "windows-1252" encoding, breaking the non-ASCII characters. I've tried setting the changing the fallback text encoding from "Default for Current Locale" (which here in the UK should be either ISO-8859-1 or UTF-8) to "Central European, ISO", "Central European, Microsoft", and "Other (incl. Western European)". None of these can display æ, ø and å. There are no Unicode options. I've also tried changing intl.fallbackCharsetList.ISO-8859-1 in about:config to various values like utf8, utf-8, iso-8859-1, with no luck.

Using this markdown package:

$ pacman --query --owns "$(which markdown)"
/usr/bin/markdown is owned by markdown 1.0.1-6

and this locale:

$ locale 
LANG=en_GB.utf8
LC_CTYPE="en_GB.utf8"
LC_NUMERIC="en_GB.utf8"
LC_TIME="en_GB.utf8"
LC_COLLATE="en_GB.utf8"
LC_MONETARY="en_GB.utf8"
LC_MESSAGES="en_GB.utf8"
LC_PAPER="en_GB.utf8"
LC_NAME="en_GB.utf8"
LC_ADDRESS="en_GB.utf8"
LC_TELEPHONE="en_GB.utf8"
LC_MEASUREMENT="en_GB.utf8"
LC_IDENTIFICATION="en_GB.utf8"
LC_ALL=

I tried to ask for a solution at the markdown command level, but that was rejected.

  • What about setting View > Text Encoding > Unicode from the Firefox menu? – Paul Nordin Sep 7 '16 at 20:20
  • @PaulNordin That's a good way to do it for one page (I had forgotten that menu even existed since they removed it by default!), but I don't think that sets the default. – l0b0 Sep 7 '16 at 20:23
  • 1
    Oh yea, it's only temporary. I think the problem might be elsewhere. Firefox relies on headers or meta to determine the file type. For example, I created a UTF-8 file test.html containing åæâéè and opened it in firefox. The output was garbage: åæâéè. However, if I add <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> to the top, it outputs properly. Not sure how you would do this for a .md file. – Paul Nordin Sep 7 '16 at 20:43
  • I also want to do this to be able to view html email from mutt by lauching firefox, still, the html file mutt produces is utf-8 without meta tags. – pepper_chico Oct 1 '16 at 5:50
4

Update: this has been fixed since Firefox 66

UTF-8-encoded HTML (and plain text) files loaded from file: URLs are now supported without <meta charset="utf-8"> or the UTF-8 BOM

https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/66#HTML


Historical information from 2016

The reasoning behind this behavior seems to be described in Mozilla bugs 815551 (Autodetect UTF-8 by default) and 1071816 (Support loading BOMless UTF-8 text/plain files from file: URLs)

As far as I understand it basically boils down to "one should always specify the encoding as detection is too unreliable".

  • For non-local content you should leverage the protocol. With HTTP this would be providing the correct charset in the Content-Type Header
  • For HTML content you may additionally use the Doctype, i.e. <meta charset="utf-8" />
  • And for anything else the only standard way left ist to specify a BOM...

Mozilla devs seem to be open for a patch that adds a preference setting, so one day it might be possible to open local BOM-less UTF-8 documents in Firefox.

4

Setting fallback encoding to UTF-8 in Firefox has been deliberately blocked - see bugzilla.mozilla.org/show_bug.cgi?id=967981#c4.

Two ways around this that I've been looking at are:

1] Apply some trivial patches to the source and build Firefox yourself to add a Unicode[UTF-8] option to Preferences|Content|Fonts & Colors|Advanced|"Fallback Text Encoding" drop-down menu.

2] Run a local [Apache] httpd server, and set up a Name-based Virtual Server, utfx, for the utf-8 encoded files in directory /my/utf-8/files. A utf-8 charset http header can then be generated, which Firefox will recognize and display the file as UTF-8 encoded. Of course, the actual file encoding has to be UTF-8!

a) /etc/httpd/httpd.conf - add:

<VirtualHost *:80>
    # This first-listed virtual host is also the default for *:80
    ServerName localhost
    DocumentRoot "/srv/httpd/htdocs"
</VirtualHost>
<VirtualHost *:80>
    ServerName utfx
    DocumentRoot "/my/utf-8/files"
      <Directory "/my/utf-8/files">
          Options Indexes
          Require all granted
      </Directory>
## show UTF-8 characters in file names:
    IndexOptions Charset=UTF-8
## for files with extension html or txt:
    AddCharset UTF-8 txt html
## for extensionless files:
      <Files *>
          ForceType 'text/plain; charset=UTF-8'
      </Files>
      <Files *\.*>
          ForceType None
      </Files>
</VirtualHost>

(Re)start the server - apachectl restart or apachectl graceful.

b) /etc/hosts - add the domain name for accessing the utf-8 encoded files:

127.0.0.1   utfx

The content-type info being sent by the server can be checked with wget -S <URL>:

wget -S http://utfx/test{æø,.txt,.html} 2>&1 >/dev/null | grep Content-Type

for the three file types (testæø, test.txt, test.html).
The output should be:

Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
Content-Type: text/html; charset=utf-8

c) about:config - add New|Boolean:

browser.fixup.domainwhitelist.utfx  "true"

then just enter utfx in the Firefox address bar to get the files list ..

  • +1 for the Bugzilla link. The solutions are unfortunately very much overkill for the problem. – l0b0 Jan 30 '17 at 17:08
3

As I have commented in your question I was struggling to obtain the same with the purpose of correctly displaying partial html (encoding is known but there's no meta tag for encoding) from Mutt in Firefox through Mailcap.

In the end I've figure out a command that works, and which may help you too:

  • uconv --add-signature -f %{charset} -t UTF-8 %s | sponge %s && firefox -new-tab %s & sleep 5

I've discovered that when your UTF-8 encoded file contains BOM, Firefox then assumes it's UTF-8. So I've used the uconv command to add the BOM signature. Assume that %{charset} is the input charset and %s is the filename. The sponge tool (from the moreutils package) helps changing the file inplace and the sleep is just so that Mutt doesn't delete the file before Firefox finishes loading it.

I have not found any other option to set a fallback encoding in Firefox.

2

If setting the fallback for only offline files, to UTF-8, is sufficient for you, then you can go to about:config and set the value of intl.charset.fallback.utf8_for_file to true.

(source)

0

A simple workaround is to produce a complete HTML file with an encoding declaration:

pandoc --metadata=pagetitle=Brukerveiledning --output=./brukerveiledning.html --standalone --to=html ./brukerveiledning.md

Shortened (less explicit and produces warnings):

pandoc --standalone ./brukerveiledning.md > ./brukerveiledning.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.