pandoc continues complain about non-utf8 character even though it seems there is no non-utf8 character

Question

I am trying to convert a markdown file to pdf using pandoc. Since my markdown contains Chinese characters, I use the following command to produce the pdf:

pandoc --pdf-engine=xelatex -V CJKmainfont=KaiTi test.md -o test.pdf

But pandoc complains that the file contains non-utf8 characters that it can not handle, the exact error message is:

Error producing PDF.
! Undefined control sequence.
pandoc.exe: Cannot decode byte '\xbd': >Data.Text.Internal.Encoding.streamDecodeUtf8With: Invalid UTF-8 stream

According to what I have find in the internet. This is largely due to the encoding of the markdown file and may have nothing to do with pandoc. My file contains a lot of chinese characters and English characters. I have converted it to utf-8 encoding.

Things I have tried but without success

I tried to transfer my file to my CentOS server and find where the invalid characters are or just remove the invalid characters. But without success.

Grep for the non-utf8 character

Following the instruction here and here(In fact, I have tried multiple top answers in the two post, but they do not work). I have verified that the system locale is set to UTF-8, output of localectl status is:

   System Locale: LANG=en_US.UTF-8
       VC Keymap: us
      X11 Layout: us

I tried to grep for non-utf8 character. Command used is grep -axv '.*' test.md. But the command output nothing. (I thought that means there are no invalid characters which can not be decoded by utf-8.)

Try to discard invalid characters

I followed the instruction here trying to remove non-utf8 characters from my file. The command I use is:

iconv -f utf-8 -t utf-8 -c test.md > output.md

After that, When I tried to convert output.md to pdf using pandoc. I still met the same error message, which suggests that the file still contains non-utf8 characters.

My question

I am surprised that the above methods does not work. How can I pinpoint which part of file is causing the problem or how to really remove the non-utf8 character from the file so that I can compile it without error?

Other information

• You can find the markdown file here.

• If you are using Linux system, you may need to set CJKmainfont to other valid Chinese font name in your system.

• on Linux system, it seems the the command to produce pdf from markdown with Chinese text should be (change the font to the valid font):

  pandoc --latex-engine=xelatex -V CJKmainfont=KaiTi test.md -o test.pdf

Answer 1

Ok, after long hours of wrestling with the problems and digging. I finally find the root cause of the problem.

The cause

The problem is that in the test.md, texts starting with backslash exist in several places which should really be taken as literals. For example,

* 一般现在时\过去时\将来时，simple present\past\future
* 现在(过去\将来)进行时，present(past\ future) continuous
* 现在（过去\将来）完成时，present(past\future) perfect
* 现在（过去\将来）完成进行时，present(past\future) perfect continuous

Backslashes in the above paragraph are just intended as a separator for different situations. It is valid markdown. But they are unfortunately processed as command by pandoc.

Solution

Use the following command instead:

pandoc -f markdown-raw_tex --pdf-engine=xelatex -V CJKmainfont=KaiTi test.md -o test.pdf

Or warp the text starting with backslash using backticks (but this is not always desired) or just use two backslashes.

Some thought

The error message from Pandoc is misleading as the problem is not related to UTF-8 decoding. I have no idea why the error message is like that.

Also, it seems that the error messages for this issue is not consistent. For example, for the above text containing backslashes. If you compile it using

pandoc -f markdown --pdf-engine=xelatex -V CJKmainfont=KaiTi test.md -o test.pdf

The error message will be something like:

Error producing PDF.
! Undefined control sequence.
l.75 一般现在时\过去时

Then it will be much easier to find where the problem is instead of digging up about utf-8 related problems.

Follow-ups

This is indeed a bug in xelatex. It may produce invalid utf-8 bytes when it encounters invalid control sequences. But pandoc just assumes that what it receives is valid utf-8 sequence. Hence the error. For a more detailed explanations, see this post.

update 2017.12.29
With the release of Pandoc 2.0.6, this behaviour is handled more properly:

Allow lenient decoding of latex error logs, which are not always properly UTF8-encoded

Now, it is easier to debug this kind of issues.

Answer 2

pandoc is complaing about byte \xbd (hexadecimal "bd"), so grep for that. e.g.

grep -n $'\xbd' file 

e.g. if I create a small file with 4 lines, one of which contains the \xbd character:

a
b
c½
d

then grep -n will tell me it's on line 3:

$ grep -n $'\xbd' file 
3:c½

NOTE: the $'\xbd' requires a unix shell like bash. See man bash and search for "QUOTING" for details.

---

BTW, the \xbd character is an extended ascii character. It may be a broken unicode sequence (many unicode characters have 0xbd as one of their byte values). On my screen it displays as a '1/2' fraction. Here's what ascii has to say about it:

$ ascii bd
ASCII 11/13 is decimal 189, hex bd, octal 275, bits 10111101: meta-=