How to replace special characters in BibTeX (e.g. {\'{o}} = ó) within a shell script?

Question

Does anyone know of any utilities or have experience converting special characters in a BibTeX file to its proper unicode equivalent in a bash script? Trouble is I can't even find a full list of the BibTeX special characters anywhere. I created a monster of a regex and I'm still encountering issues with escaping because the particular way BibTeX special characters (ie with backslashes, quotes, and braces) is not very compatible with RegEx or shell code...

I was thinking perhaps it might be enough to have a simple regex with a capture group that looks for the {\ } pattern within a BibTeX tag, and if there is a match, the contents of that (e.g. {\\'{o}}) could be matched using some kind of external dictionary file I create, to map that sequence to its proper unicode character... ó in this case. It's just hard to do this without having access to a complete list of the special characters. And they tend to be very common in research paper metadata (what I am processing).

Also another issue is that different BibTex files will have different syntax for the special characters - the name Müller might be encoded as author = {M{\"{u}}ller}, author = {M{\"{\u}}ller} (extra backslash in the inner braces), or even author = {M\"{u}ller} (no braces around special character TeX fragment).

I have an interactive shell script that lets me process research papers I've downloaded, view/skip/forget them, or enter their DOI, at which point the shell script retrieves the paper's BibTeX, renames, the file according to a convention I use, using the BibTex tag data and DOI, and then after I'm done with the session it imports all of the processed files into Zotero for me and then runs another script I wrote that adds the both PDF attachments and Dropbox links for each paper processed. But these special characters are gunking up my filenames and my Zotero metadata...

TL;DR - Does anyone know how to convert BibTeX/TeX special characters to unicode within the context of a bash script? Or does anyone know where I can find a complete list of BibTeX special characters?

Answer 1

I think Python is better-suited than Bash for working with strings and Unicode. You can use it easily within Bash, and with this simple template, you don't need to know Python:

#!/usr/bin/env python3
import sys
for line in sys.stdin:
    line = line.replace(r"\'{o}",r"ó")
    line = line.replace(r'{\"{u}}',r"ü")
    line = line.replace(r'{\"{\u}}',r"ü")
    line = line.replace(r'\"{u}',r"ü")
    print(line, end='')

Usage:

• copy-paste the above into a new text file, e.g. convert_tex_chars.py
• open a terminal and cd to the directory where the text file is
• make the file executable: chmod +x convert_tex_chars.py
• test: echo "s\'{o}ng p\'{o}ng "'M{\"{u}}ller M{\"{\u}}ller M\"{u}ller' |./convert_tex_chars.py (this should display sóng póng Müller Müller Müller)
• test it on your BibTeX file: cat original_bibtex_file |./convert_tex_chars.py >new_bibtex_file

Edit the Python file and add or edit replacements as needed. Note that, in Python, " and ' work the same, so use "" if the contents contains a single quote and '' if the contents contains a double quote. Note that the r before each string allows you to include the backslash character without escaping it (more details).

If you want regex substitutions, insert import re as the 2nd line, and then use it like this: line = re.sub(r'\\', r'\\', line) (this replaces \\ with \). The first parameter is your regex, the second is the substitution (backslashes are processed here), and the third is the source string. See the official docs and hundreds of examples around the web.