0

Does anyone know of any utilities or have experience converting special characters in a BibTeX file to its proper unicode equivalent in a bash script? Trouble is I can't even find a full list of the BibTeX special characters anywhere. I created a monster of a regex and I'm still encountering issues with escaping because the particular way BibTeX special characters (ie with backslashes, quotes, and braces) is not very compatible with RegEx or shell code...

I was thinking perhaps it might be enough to have a simple regex with a capture group that looks for the {\ } pattern within a BibTeX tag, and if there is a match, the contents of that (e.g. {\\'{o}}) could be matched using some kind of external dictionary file I create, to map that sequence to its proper unicode character... ó in this case. It's just hard to do this without having access to a complete list of the special characters. And they tend to be very common in research paper metadata (what I am processing).

Also another issue is that different BibTex files will have different syntax for the special characters - the name Müller might be encoded as author = {M{\"{u}}ller}, author = {M{\"{\u}}ller} (extra backslash in the inner braces), or even author = {M\"{u}ller} (no braces around special character TeX fragment).

I have an interactive shell script that lets me process research papers I've downloaded, view/skip/forget them, or enter their DOI, at which point the shell script retrieves the paper's BibTeX, renames, the file according to a convention I use, using the BibTex tag data and DOI, and then after I'm done with the session it imports all of the processed files into Zotero for me and then runs another script I wrote that adds the both PDF attachments and Dropbox links for each paper processed. But these special characters are gunking up my filenames and my Zotero metadata...

TL;DR - Does anyone know how to convert BibTeX/TeX special characters to unicode within the context of a bash script? Or does anyone know where I can find a complete list of BibTeX special characters?

4
  • 2
    your question is best suitable (I think) to ask on tex.stackexchange.com; Re: does anyone know where I can find a complete list of BibTex special characters, check official website bibtex.org/SpecialSymbols – αғsнιη Jan 21 at 10:36
  • 1
    @αғsнιη thanks, I considered that, but the fact is I am trying to parse the BibTeX within a shell script, and I would prefer the aid of a shell expert who knows a little about TeX than an TeX expert who knows a little about shell scripting, if that makes sense… – Avana Vana Jan 21 at 11:40
  • One of the problems is that contributors usually would like to see what you already tried, and where you ran into difficulties, in order to see what kind of tools you have at your hand, understand whether there are special conditions that need consideration, and also to discourage the occasional user who considers this site a "free script-writing service". Perhaps you can edit your post to include that. – AdminBee Jan 21 at 15:30
  • I want to also note that I found the additional resources that helped me create the exhaustive list of BibTeX special characters I linked in a comment to the "Answer" below. en.wikibooks.org/wiki/LaTeX/Special_Characters#Escaped_codes , giss.nasa.gov/tools/latex/ltx-401.html , ctan.org/tex-archive/info/bibtex/tamethebeast – Avana Vana Jan 22 at 23:39
0

I think Python is better-suited than Bash for working with strings and Unicode. You can use it easily within Bash, and with this simple template, you don't need to know Python:

#!/usr/bin/env python3
import sys
for line in sys.stdin:
    line = line.replace(r"\'{o}",r"ó")
    line = line.replace(r'{\"{u}}',r"ü")
    line = line.replace(r'{\"{\u}}',r"ü")
    line = line.replace(r'\"{u}',r"ü")
    print(line, end='')

Usage:

  • copy-paste the above into a new text file, e.g. convert_tex_chars.py
  • open a terminal and cd to the directory where the text file is
  • make the file executable: chmod +x convert_tex_chars.py
  • test: echo "s\'{o}ng p\'{o}ng "'M{\"{u}}ller M{\"{\u}}ller M\"{u}ller' |./convert_tex_chars.py (this should display sóng póng Müller Müller Müller)
  • test it on your BibTeX file: cat original_bibtex_file |./convert_tex_chars.py >new_bibtex_file

Edit the Python file and add or edit replacements as needed. Note that, in Python, " and ' work the same, so use "" if the contents contains a single quote and '' if the contents contains a double quote. Note that the r before each string allows you to include the backslash character without escaping it (more details).

If you want regex substitutions, insert import re as the 2nd line, and then use it like this: line = re.sub(r'\\', r'\\', line) (this replaces \\ with \). The first parameter is your regex, the second is the substitution (backslashes are processed here), and the third is the source string. See the official docs and hundreds of examples around the web.

7
  • Thanks for your response. I actually did something similar but without Python. I am just using awk to replace instances of special characters in BibTeX fields as I parse them in my bash script, using this tab-separated dictionary I created. Took a while due to the sheer variety of BibTeX syntax out there in real research paper .bib files. gist.github.com/avanavana/e52fc427e7bd72a5532ffe16d45d157d – Avana Vana Jan 22 at 15:39
  • Your method actually is probably the simplest. I mean, it results in a fairly large chain of replace calls (gist.github.com/avanavana/c035d971e23bf4973d838b2650bcff56), but performance isn't bad and I am using this script manually, interactively, and the service that provides me with the BibTeX requires rate limiting anyway... so, I think I'll just go with this. The toughest part for me here is how to make sure the original string is quoted correctly, since BibTeX special chars use both single and double quotes. I can't predict what the BibTeX I get from curl is going to use. – Avana Vana Jan 22 at 21:06
  • That is a large chain of replace calls. You could put the replacements in a text file and loop through them in the Python program, but I'm not sure it would be any faster. I'm glad you found a system that works for what you need. – bitinerant Jan 22 at 21:16
  • @AvanaVana - I should have mentioned this before, but there is another way to do quoting in Python - triple quotes, e.g. line = line.replace(r"""{\'{g}}""",r"""ǵ"""). You can use this for everything, i.e. include both single and double quotes within it, so long as you don't have any triple quotes within the string. – bitinerant Jan 22 at 21:40
  • I know... not ideal, but surprisingly quick, and as I mentioned, this is an interactive shell script which involves the manual step of me finding the DOI of a given research paper, either by just copying it from the paper itself (the script opens the file for me to review), or by finding it online (not all papers include the DOI in the text). I had previously written a script that extracted DOIs from all PDFs that had one in their kMDItem metadata, but surprisingly only a handful actually had this metadata, and they vary too much to try to parse PDF body text automatically… (1/2) – Avana Vana Jan 22 at 23:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.