I used OCR to turn some scans into plaintext, but unfortunately the letters 'fi' which are commonly joined in some fonts, got read in as capital W's. Now I need to replace all the W's with 'fi', and these can easily be distinguished by the fact that a capital W does not ever occur in the middle of a word in true English. So, I need a sed one-liner that replaces all word-medial capital W's with the letters fi.
A capital W doesn't occur at the end of a word either, but it may occur in an all-caps abbreviation. So I'd replace
W when it's immediately after a lowercase letter, or when it follows an uppercase letter and precedes a lowercase letter (aWre).
sed -e 's/\([[:lower:]]\)W/\1fi/g' -e 's/\([[:alpha:]]\)W\([[:lower:]]\)/\1fi\2/g'
This doesn't cover
fifi (which my biggest word list only finds it in “fifing”). More importantly, this doesn't cover
W at the beginning of a word; you can capture some cases by looking at the second letter, but that's still going to miss many words that begin with
fi. In English, many letters never appear after a W:
… -e 's/\([^[:alnum:]]\)W\([b-dfgj-npqstv-xz]\)/\1fi\2/g' \ -e 's/^W\([b-dfgj-npqstv-xz]\)/fi\2/'
For more precise results and to cope with other languages, you can switch to a more complex dictionary-based approach (which fancy OCR systems often use, evidently yours isn't fancy enough).
A quick and simple answer that should solve most cases is:
sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/"
This will replace any
W that is does not have a space before or after it with
fi. It catches the error in these simple test cases:
$ echo "blah blah blah trafWc" | sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/" blah blah blah traffic $ echo "blah blah blah Wallaby" | sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/" blah blah blah Wallaby
However, it won't capture things such as:
$ echo "blah blah blah Wnger" | sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/" blah blah blah Wnger
As you can see this should have changed to finger. So unfortunately you'll still need to check for those.
The guys have already covered the basic
sed stuff, but there are a couple more things you could do to help your text along.
The first is to train your OCR program to recognize these in the first place. Most OCR systems have some sort of system to teach it new letters and learn from common mistakes like this one. If your corpus is big enough to need to to search and replaces like this, it should be big enough to teach the OCR engine not to make that mistake.
Secondly, you can run your text through a spell check engine. Many OCR programs include this step as part of an internal self check, a sort of "does this word even make sense transcribed this way". You can certainly do this yourself as a way of validating that everything got sanely converted.