3

I want to create a sed command that will remove all of these strange characters from a given document:

sed -n 's/\|®MD-IT¯\|®MD\+BO¯\|®MDNM¯®LL\.8LI,0LI¯\|®LL0LI,0LI¯\|®MD\+IT¯\|®LL.8LI,0LI¯®MDIT¯\|®MDNM¯®FL¯®LL.8LI,0LI¯\|®FL¯®MD-BO¯\|®FL¯®MD-BO¯\|®MD-BO¯\|¯®OF1IN,1IN¯®FC¯®LL1LI,0LI¯\|\|®SF1,1¯\|®FM1FT=0LI,LR=1;\|®MDSU¯®FN1¯\|®MDNM¯¯\|®IV-RTF\|\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\|¯®BF0¯\|®FS1\|-------------------------------------\|¯®FW1\|\|//gp'

These codes were all created in another application Nota Bene and I have many files with such codes that I would like to convert to plain text and possibly even markdown.

The problem is that the characters are not substituted. I have tried doing this in Sublime Text and was successful in stripping the document using find-replace (regex). It would be better for me to create a sed script than to use Sublime for this task.

I also tried using Ed but it too did not pick up the replacements.

Here is a sample nb file when opened in `Sublime Text:

®SSDEFAULTS¯®LR1¯®JU¯®MD+BO¯®UFTimes New Roman¯®SZ12Pt¯Glossary®MD+BO¯®TS.5IN,1IN,1.5IN,2IN,2.5IN,3IN,3.5IN,4IN,4.5IN,5IN,5.5IN,6IN¯    ®MD-BO¯
®NJ¯®LR1¯®LL.5LI,0LI¯®MD+BO¯®LL0LI,0LI¯®MDNM¯®LR1¯®LL.5LI,0LI¯A fortiori proposition: If X is true, then how much greater is Y true? To move logically from a stronger argument to establish a weaker argument. The weaker argument is sometimes presented by the speaker as the stronger argument.
®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯Accusative of motion/direction - Indicates movement to the noun marked by the accusative and is to be distinguished from the accusative of local determination which indicates location without motion (Joüon and Muraoka 2006, 428).
Anadiplosis - A figure of speech in which the word that a colon ends with, or a like sounding word, is the word that begins the next colon ®GC|CI:R#=47;AU=Brown, Raymond E.;YR=1990;TI=New Jerome biblical commentary;PG=245;XT=;F[=;F]=;F#=;ID=;XX=Print;CT=;FL=¯(Brown, Fitzmyer, Murphy, et al. 1990, 245)®GC¯.
®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯Anaphoric use of the article - When the article is used to indicate that the word to which it is attached is the one previously mentioned (Williams and Beckman 2007, 36). 
®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯Anaptyxis - The insertion of a vowel into a word to avoid a consonant cluster.
®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯Aoristic perfect - I use the phrase 'aoristic perfect' to refer to one of the ways the qatal form can be rendered into English. Aoristic perfect denotes a past situation the implications of which are no longer felt in the present. The situation may have extended over a period of time and it may have occurred more than once. It may have occurred in the recent or distant past but from the standpoint of the speaker it is to be regarded as a fact having occurred and hence as a fact belonging to the past (Joüon and Muraoka 2006, 337; Driver 1998, 12). The term 'aoristic perfect' and indeed the other categorizations of perfect in this grammar, all relate to the interpretation of qatal verbs in their given contexts. The qatal form in and of itself does not convey these meanings. 
®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯®LL0LI,0LI¯®LR1¯®LL.5LI,0LI¯Beth essentiae - ®LAHebrew¯ÿHá®LAEnglish¯ that is used to indicate the predicate of a clause or a word used predicatively (Joüon and Muraoka 2006, 458).

This is how I would like the text to read:

Glossary    
A fortiori proposition: If X is true, then how much greater is Y true? To move logically from a stronger argument to establish a weaker argument. The weaker argument is sometimes presented by the speaker as the stronger argument.
Accusative of motion/direction - Indicates movement to the noun marked by the accusative and is to be distinguished from the accusative of local determination which indicates location without motion (Joüon and Muraoka 2006, 428).
Anadiplosis - A figure of speech in which the word that a colon ends with, or a like sounding word, is the word that begins the next colon (Brown, Fitzmyer, Murphy, et al. 1990, 245).
Anaphoric use of the article - When the article is used to indicate that the word to which it is attached is the one previously mentioned (Williams and Beckman 2007, 36). 
Anaptyxis - The insertion of a vowel into a word to avoid a consonant cluster.
Aoristic perfect - I use the phrase 'aoristic perfect' to refer to one of the ways the qatal form can be rendered into English. Aoristic perfect denotes a past situation the implications of which are no longer felt in the present. The situation may have extended over a period of time and it may have occurred more than once. It may have occurred in the recent or distant past but from the standpoint of the speaker it is to be regarded as a fact having occurred and hence as a fact belonging to the past (Joüon and Muraoka 2006, 337; Driver 1998, 12). The term 'aoristic perfect' and indeed the other categorizations of perfect in this grammar, all relate to the interpretation of qatal verbs in their given contexts. The qatal form in and of itself does not convey these meanings. 
|> sed -n l Glossary.NB
\256SSDEFAULTS\257\256LR1\257\256JU\257\256MD+BO\257\256UFTimes New R\
oman\257\256SZ12Pt\257Glossary\256MD+BO\257\256TS.5IN,1IN,1.5IN,2IN,2\
.5IN,3IN,3.5IN,4IN,4.5IN,5IN,5.5IN,6IN\257\t\256MD-BO\257\r$
\256NJ\257\256LR1\257\256LL.5LI,0LI\257\256MD+BO\257\256LL0LI,0LI\257\
\256MDNM\257\256LR1\257\256LL.5LI,0LI\257A fortiori proposition: If X\
 is true, then how much greater is Y true? To move logically from a s\
tronger argument to establish a weaker argument. The weaker argument \
is sometimes presented by the speaker as the stronger argument.\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Accusative of motion/direction - Indicates mov\
ement to the noun marked by the accusative and is to be distinguished\
 from the accusative of local determination which indicates location \
without motion (Jo\374on and Muraoka 2006, 428).\r$
Anadiplosis - A figure of speech in which the word that a colon ends \
with, or a like sounding word, is the word that begins the next colon\
 \256GC|CI:R#=47;AU=Brown, Raymond E.;YR=1990;TI=New Jerome biblical \
commentary;PG=245;XT=;F[=;F]=;F#=;ID=;XX=Print;CT=;FL=\257(Brown, Fit\
zmyer, Murphy, et al. 1990,\240245)\256GC\257.\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Anaphoric use of the article - When the articl\
e is used to indicate that the word to which it is attached is the on\
e previously mentioned (Williams and Beckman 2007, 36). \r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Anaptyxis - The insertion of a vowel into a wo\
rd to avoid a consonant cluster.\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Aoristic perfect - I use the phrase 'aoristic \
perfect' to refer to one of the ways the qatal form can be rendered i\
nto English. Aoristic perfect denotes a past situation the implicatio\
ns of which are no longer felt in the present. The situation may have\
 extended over a period of time and it may have occurred more than on\
ce. It may have occurred in the recent or distant past but from the s\
tandpoint of the speaker it is to be regarded as a fact having occurr\
ed and hence as a fact belonging to the past (Jo\374on and Muraoka 20\
06, 337; Driver 1998, 12). The term 'aoristic perfect' and indeed the\
 other categorizations of perfect in this grammar, all relate to the \
interpretation of qatal verbs in their given contexts. The qatal form\
 in and of itself does not convey these meanings. \r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Beth essentiae - \256LAHebrew\257\377H\341\256\
LAEnglish\257 that is used to indicate the predicate of a clause or a\
 word used predicatively (Jo\374on and Muraoka 2006, 458).\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Classic perfect - I use the phrase 'classic pe\
rfect' to refer to one of the ways the qatal form can be rendered int\
o English. Classic perfect refers to the continuing present relevance\
 of a past situation from the perspective of the speaker (Comrie 1976\
, 52). By perfect I do not necessarily imply that a previous situatio\
n has resulted in a state but that the situation has implications rel\
evant to the present. The situation is not merely past and over but s\
omehow persists and continues to intrude into the present. Such verbs\
 are usually translated into English using the perfect or present ten\
se. I have included under this definition quasi-stative verbs which r\
efer to attributes which were acquired before, but which are assumed \
to continue in some way up to the present moment (Driver 1998, 11; Jo\
\374on and Muraoka 2006, 333; Waltke and O'Connor 1990, 487). In some\
 grammars these are treated separately. However, that creates too man\
y functions for the one perfect form. The term 'classic perfect' and \
indeed the other categorizations of perfect in this grammar all relat\
e to the \256MD+IT\257interpretation \256MD-IT\257of qatal verbs in t\
heir given contexts. The qatal form by itself does not convey these m\
eanings.\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Cohortative of praise. The cohortative is ofte\
n used in Psalms to indicate that praise, freely undertaken, has begu\
n. This usage is close to the cohortative of resolve but not identica\
l with it. The emphasis falls not on what the writer is intending to \
do, but what he has already undertaken. \r$
Cohortative of resolve - The cohortative mood normally expresses the \
will of the speaker, but when the speaker has the ability to carry ou\
t what he wants it takes on the coloring of resolve (Van der Merwe et\
 al. 1997, 152; Waltke and O'Connor 1990, 573).\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Concluding \256LAHebrew\257\377h\353\377H\351\
\256LAEnglish\257 - A special use of the word \256LAHebrew\257\377h\
\353\377H\351\256LAEnglish\257 found towards the end of several Psalm\
s and approximating in meaning to: the conclusion of the matter is th\
at\205\r$
\256LL0LI,0LI\257\256LR1\257\256LL.5LI,0LI\257\256LL0LI,0LI\257\256LR\
1\257\256LL.5LI,0LI\257Conjunctive waw - Waw used to connect clauses \
10
  • 1
    You seem to be using extended regular expressions. Does it work if you use sed with -E (and change the \| to | if it's alternations you want)?
    – Kusalananda
    Jun 29 at 6:59
  • 2
    FYI, notabene.com has a 30-day trial version available for download (Windows or Mac. Windows version should run in WINE, they explicitly mention it). It says printing is disabled, doesn't mention export. If export still works in the trial version, you can export .nb files to Rich Text Format (RTF) - unfortunately, this will probably be a manual point & click GUI bs operation for each .nb file. You can convert RTF to markdown with pandoc, or to LaTeX or plain text with unrtf. This final step should be easily scripted.
    – cas
    Jun 29 at 10:25
  • 2
    It's unclear what "the codes" are. Do they include the pipe symbols, dots and pluses, and /or the backslashes? Can you show an example document, and what you want it to look like after transformation? You are running the sed command with -n, and then with the p flag for the substitution command, indicating that you only want to see lines that were modified. Is that correct?
    – Kusalananda
    Jun 29 at 11:37
  • 2
    Could you do something like sed -n l file.nb and paste the output (or part of it) in the question ? So we can see the actual values of the unwanted characters. Jun 29 at 13:06
  • 2
    Interestingly, you could probably get enough information to write a decent parser/converter just by running grep -o '®[^¯]*[¯]' *.nb | sort -u. Just with the sample shown, I can guess that ®GC(.*)®GC¯ is some sort of biblio format (containing KEY=VALUE pairs), ®JU¯ is justfication on, ®NJ¯ is justify off, ®UF sets the font, ®SZ sets the font size, ®LA sets the language, ®TS(.*)¯ sets the tab stops, etc. LL & LI probably set the line spacing (or maybe left margin) and indenting. Dunno what LR, MD, NM, or BO are for - but they probably make sense if you're used to using Nota Bene.
    – cas
    Jun 29 at 14:05

3 Answers 3

6

Sed can also be used as a script (easier to devel): create a file "nb2txt" with

#!/usr/bin/sed -Ef

s/®[^¯]*¯//g
s/-{20,}//g
s/\.{20,}//g

and:

$ chmod 755 nb2txt
$ nb2txt file.nb
3
  • 2
    +1. I was going to post a comment about simplifying most of the sed script to just sed -e 's/®[^¯]*[¯]//g' input.txt > out.txt but your answer already does that. This does delete the weird bibliiography format on the fourth input line, but that could be handled separately with an earlier rule dealing with ®GC\|(.*)FL=¯ if required...the OP doesn't seem to mind it being deleted but that could be because they think it's too hard to bother with (probably is in sed, definitely not in perl -p)
    – cas
    Jun 29 at 13:40
  • 1
    @cas, moving to Perl would obviously be my next step! (and is trivial (even in sed) to process some of the codes and build an useful and improvable nb2md tool.
    – JJoao
    Jun 29 at 14:05
  • 2
    minor correction: i meant s/®[^¯]*¯// - the square brackets around the second ¯ are a remnant from when i was trying to avoid deleting the bibliographic data on line 4 with '®[^¯=]*[¯=]'
    – cas
    Jun 29 at 14:48
3

Your regular expression uses \| (alternate pattern in GNU sed, literal bar in most other sed implementations) and \+ (one or more occurrences in GNU sed, literal + in most other sed implementations). If you use GNU sed, this pattern will delete any pattern like ®MD-IT¯ or ®MDDDDDBO¯. If you are using a different sed implementation, it will probably not find any match.

Better use extended regular expressions, supported by most sed versions for years:

sed -nE 's/®MD-IT¯|®MD+BO¯|®MDNM¯®LL\.8LI,0LI¯|®LL0LI,0LI¯|… and so on

I also suggest to remove the empty alternatives (\| at the start and end of the pattern), although they don't do harm in this case.

And the endless \.\.\.\.\.\.\.\.\.\.\.\. and ---- should be replaced by \.{42} and -{23} with the actual number of dots or dashes. Or maybe by \-{10,} to get rid of any occurrence of 10 or more dots.

1
  • I am still testing this.
    – Edman
    Jun 29 at 17:31
1

From the sed -n l listing it is clear that you have a file with many characters 174 (in decimal or 256 in octal) and [character 175] (in decimal), or 257 (in octal). Listed as \256 and \257 and that may be interpreted as Unicode \xae (Hexadecimal code ae -or 256 in octal-) or just ®, if interpreted as a "one byte" character, and Unicode \xaf (Hexadecimal code af -or 257 in octal) or just ¯, if interpreted as a single byte character,

$ printf '\256 \257 \n' | iconv -f WINDOWS-1252 -t utf8
® ¯

If you use utf8 as the default encoding (the usual in Linux).

And which seem to start and end some internal encoding of the .nb files. Removing strings that start with \xae and end with \xaf seems to get us one step closer to your request:

$ sed 's/®[^¯]*¯//g' test
Glossary    
A fortiori proposition: If X is true, then how much greater is Y true? To move logically from a stronger argument to establish a weaker argument. The weaker argument is sometimes presented by the speaker as the stronger argument.
Accusative of motion/direction - Indicates movement to the noun marked by the accusative and is to be distinguished from the accusative of local determination which indicates location without motion (Joüon and Muraoka 2006, 428).
Anadiplosis - A figure of speech in which the word that a colon ends with, or a like sounding word, is the word that begins the next colon (Brown, Fitzmyer, Murphy, et al. 1990, 245).
Anaphoric use of the article - When the article is used to indicate that the word to which it is attached is the one previously mentioned (Williams and Beckman 2007, 36). 
Anaptyxis - The insertion of a vowel into a word to avoid a consonant cluster.
Aoristic perfect - I use the phrase 'aoristic perfect' to refer to one of the ways the qatal form can be rendered into English. Aoristic perfect denotes a past situation the implications of which are no longer felt in the present. The situation may have extended over a period of time and it may have occurred more than once. It may have occurred in the recent or distant past but from the standpoint of the speaker it is to be regarded as a fact having occurred and hence as a fact belonging to the past (Joüon and Muraoka 2006, 337; Driver 1998, 12). The term 'aoristic perfect' and indeed the other categorizations of perfect in this grammar, all relate to the interpretation of qatal verbs in their given contexts. The qatal form in and of itself does not convey these meanings. 
Beth essentiae - ÿHá that is used to indicate the predicate of a clause or a word used predicatively (Joüon and Muraoka 2006, 458).
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.