0

I am writing a shell script to convert some DokuWiki pages to MediaWiki format and vice versa. I'm having some troubles with footnotes.

There is a plugin for DokuWiki that adds extended features to DokuWiki's basic footnote markup. One of them is the ability to add a name to a note and reuse it later. For example:

DokuWiki MediaWiki
[(FOO>This is a footnote.)] <ref name="FOO">This is a footnote.</ref>
[(BAR>Another note in the same paragraph.)] <ref name="BAR">Another note in the same paragraph.</ref>

With sed it is easy to find and replace. This is the command of my script related to footnotes with "note names":

sed -ri 's@\[\(.*>@<ref name=\"XXX\">@g' dokuwiki-page.txt
sed -ri 's@\)\]@<\/ref>@g' dokuwiki-page.txt

But, of course, it doesn't preserve the names, it just applies a the new generic note name "XXX" to all notes that has a name. So, in my example above, the result will be:

DokuWiki MediaWiki
[(FOO>This is a footnote.)] <ref name="XXX">This is a footnote.</ref>
[(BAR>Another note in the same paragraph.)] <ref name="XXX">Another note in the same paragraph.</ref>

I need help to preserve the note name (FOO and BAR in the example). I do accept other solutions than just sed.

IMPORTANT NOTES:

  1. The footnote sentence can appear in the middle of a paragraph, and more than one reference with a footnote name can appear on the same paragraph but with a different name. (paragraph a.k.a. Unix's "very long lines")
  2. I can't split the command to replace [( in a first command and > later in a second command because MediaWiki markup makes use of too many html tags (full of < and >). One tag can be replaced improperly.
  3. There are some [(...)]s without > inside. Like in [(This is a nameless note.)] as opposed to [(My_Note_Name>This is a named note.)].

2 Answers 2

3

This kind of thing is much easier with perl whose regexps have non-greedy repetition operators:

perl -i -pe 's{\[\((.*?)>(.*?)\)\]}{<ref name="$1">$2</ref>}g' your-file

Note that -i and -r are non-standard sed options. -i was actually copied from perl by a few implementations though in incompatible fashion between each other.

perl, contrary to several sed implementations also has no limit on the size of lines, can cope with NUL characters and by default treats the input byte-wise so doesn't have issues with input that can't be decoded as text in the user's locale.

If your input can have some [(...)] that don't contain >s, then you'll need to adapt the regexp. If the reference tags only contain word characters (ASCII alnums and underscores), then it could be just:

perl -i -pe 's{\[\((\w+)>(.*?)\)\]}{<ref name="$1">$2</ref>}g' your-file

Another approach is to find all the [(...)] and do the replacements within as a separate step:

perl -i -pe '
  s{\[\(.*?\)\]}{
    $& =~ s{\[\((.*?)>(.*)\)\]}{<ref name="$1">$2</ref>}r
  }ge' your-file

That would also allow use to change the nameless notes to <ref>nameless</ref> with:

perl -i -pe '
  s{\[\(.*?\)\]}{
    $& =~ s{\[\((?:(.*?)>)?(.*)\)\]}{
      "<ref" . (defined($1) ? qq( name="$1") : "") . ">$2</ref>"
    }re
  }ge' your-file

Or to use negative look-ahead operators to make sure the things that are matched inside the [(...)] don't include )]:

perl -i -pe 's{\[\(((?:(?!\)\]).)*?)>((?1))\)\]}{<ref name="$1">$2</ref>}g' your-file
1
  • Very good, but also have the same problem the sed solution i posted before. Thank you for your help!
    – Daniel
    Commented Dec 3, 2023 at 17:02
0

Definitive SED way:

I found a solution by using sed and regex groups.

sed -Ei 's@\[\(([[:alnum:]_-]*)>([[:alnum:][:space:].!?:;,@#%$&<>-_]*)\)\]@<ref name=\"\1\">\2<\/ref>@g' dokuwiki-page.txt

Explanation:

  1. Find lines with [( + letters, numbers, underscores and dashes in any quantity + > + letters, numbers, spaces and punctuation + )]
    • group 1: letters, numbers, underscores and dashes in any quantity.
    • group 2: letters, numbers, spaces and most of punctuation signals in any quantity. For some reason, [:punct:] doesn't work well and i should use a big list: .!?:;,@#%$&<>-_.
    • The trick here is that groups can be referenced using \1 or \2. It's like storing it in a variable.
    • I can't use .* instead of ([[:alnum:]_-]*) because it includes other >. So if there are any other named footnotes in the same paragraph (aka very long lines), the regex will include everything from the first footnote to the end of the second footnote. A mess!
  2. Replace all this by <ref name=" + group \1 + "> + group \2 + </ref>.
    • Here I used \1 and \2 to back reference what i want to keep while replacing what was around it.

Very, very hard! It took me three days of research to figure out how to do this. And it's so long. Better go for perl. But please, if you know an easier way with sed teach me, i like to learn!

Reading suggestion:

  • DOUGHERTY, D., & ROBBINS, A. (1997). SED & AWK. (2nd ed.). O’Reilly.
4
  • Uh, of course, need another command. But it's a definitive solution for SED only, i accept your solution too, it's very good!
    – Daniel
    Commented Dec 4, 2023 at 14:07
  • 1
    You can apply many editing commands in a single sed call if you just use -e: sed -e expr1 -e expr2 -e expr3. This way you will not have broken XML in your file in-between the commands.
    – Kusalananda
    Commented Dec 4, 2023 at 14:20
  • 1
    That second sed would also replace the )]'s in [(This is a nameless note.)]. Commented Dec 4, 2023 at 14:25
  • Fixed! Edited above. Thank you for all your help!
    – Daniel
    Commented Dec 8, 2023 at 1:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .