1

I am debugging my tex file by eliminating all technical flaws in the systems. I cannot find anything wrong in my document with Tex community here and myself so I think there can be something non-ASCII characters complicating pdflatex. Sample of LaTeX warnings which me and my friends do not understand, since we do not understand the origin

Underfull \hbox (badness 1019) in paragraph at lines 4--10
\T1/lmr/m/n/12 (+20) Avail-able from [][]$\T1/lmtt/m/n/12 http : / / www . dyna
med . com / login . aspx ? direct = true & site =
[50] [51] [52] [53]
Underfull \hbox (badness 1019) in paragraph at lines 332--338
\T1/lmr/m/n/12 (+20) Avail-able from [][]$\T1/lmtt/m/n/12 http : / / www . dyna
med . com / login . aspx ? direct = true & site =
) (./deliminated.figures.tex [54]

Things which I have done to try to eliminate all non-British non-ASCII characters in .tex and .bib files

  1. Eliminate non-ASCII characters in .tex and .bib

    # http://stackoverflow.com/a/9395552/54964
    grep --color='auto' -P -n "[\x80-\xFF]" file.tex
    grep --color='auto' -P -n "[\x80-\xFF]" work.bib
    
  2. ...

I am interested in if the grep is sufficient for searching all non-ASCII characters.

Tools tested but no benefit after Gilles' answer

  1. grep --color='auto' -P -n '[^\x00-\x7F]' file.tex
  2. grep -P "[\x80-\xFF]" file.tex
  3. perl -ane '{ if(m/[[:^ascii:]]/) { print } }' file.tex
  4. grep --color='auto' -P -n "[^[:ascii:]]" file.txt

Things I am not sure

  1. Gilles' answer here LC_ALL=C grep '[^ -~]' file.tex but it finds LaTeX syntax characters as non-ASCII. Is this right?

TeXLive: 2016
OS: Debian 8.5
Locale: British

  • you may want to prefix that with LC_ALL=C, so LC_ALL=C grep --color='auto' -P -n "[\x80-\xFF]" file.tex – iruvar Nov 26 '16 at 20:17
3

[\x80-\xFF] only includes code points from 128 to 255. Depending on the locale, specifically on the character encoding, this may or may not be the set of non-ASCII characters. It is only the case in 8-bit encodings. (ASCII-based 8-bit encodings, to be precise, but you're extremely unlikely to encounter anything else.) In locales where a character isn't a single byte, including UTF-8 which is the de facto standard, [\x80-\xFF] is only a small subset of non-ASCII characters.

The easiest way to search for non-ASCII characters is to impose the C locale. In the C locale, a character is a byte. Using a unibyte locale also makes some versions of GNU grep considerably faster.

LC_ALL=C grep --color='auto' -P -n "[\x80-\xFF]" file.tex
  • 1
    I was going to suggest recode, and then I noticed that echo œ | recode utf8..tex produces oe. Yeech. – Gilles 'SO- stop being evil' Nov 27 '16 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.