Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I used OCR to turn some scans into plaintext, but unfortunately the letters 'fi' which are commonly joined in some fonts, got read in as capital W's. Now I need to replace all the W's with 'fi', and these can easily be distinguished by the fact that a capital W does not ever occur in the middle of a word in true English. So, I need a sed one-liner that replaces all word-medial capital W's with the letters fi.

share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

A capital W doesn't occur at the end of a word either, but it may occur in an all-caps abbreviation. So I'd replace W when it's immediately after a lowercase letter, or when it follows an uppercase letter and precedes a lowercase letter (aWre).

sed -e 's/\([[:lower:]]\)W/\1fi/g' -e 's/\([[:alpha:]]\)W\([[:lower:]]\)/\1fi\2/g'

This doesn't cover fifi (which my biggest word list only finds it in “fifing”). More importantly, this doesn't cover W at the beginning of a word; you can capture some cases by looking at the second letter, but that's still going to miss many words that begin with fi. In English, many letters never appear after a W:

… -e 's/\([^[:alnum:]]\)W\([b-dfgj-npqstv-xz]\)/\1fi\2/g' \
  -e 's/^W\([b-dfgj-npqstv-xz]\)/fi\2/'

For more precise results and to cope with other languages, you can switch to a more complex dictionary-based approach (which fancy OCR systems often use, evidently yours isn't fancy enough).

share|improve this answer
add comment

A quick and simple answer that should solve most cases is:

sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/"

This will replace any W that is does not have a space before or after it with fi. It catches the error in these simple test cases:

$ echo "blah blah blah trafWc" | sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/"
blah blah blah traffic

$ echo "blah blah blah Wallaby" | sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/"
blah blah blah Wallaby

However, it won't capture things such as:

$ echo "blah blah blah Wnger" | sed "s/\([^ ]\)W\([^ ]\)/\1fi\2/"
blah blah blah Wnger

As you can see this should have changed to finger. So unfortunately you'll still need to check for those.

share|improve this answer
1  
Two corner cases that your pattern won't match: SuW and wild-Wre (Sufi and wild-fire, the second example being broken between lines). Not sure that you can match a broken hyphenated word in a one-liner, however. Upvote anyway, as we arrived at the same pattern to get most of the job done. –  Eli Heady May 27 '11 at 1:59
add comment

The guys have already covered the basic sed stuff, but there are a couple more things you could do to help your text along.

The first is to train your OCR program to recognize these in the first place. Most OCR systems have some sort of system to teach it new letters and learn from common mistakes like this one. If your corpus is big enough to need to to search and replaces like this, it should be big enough to teach the OCR engine not to make that mistake.

Secondly, you can run your text through a spell check engine. Many OCR programs include this step as part of an internal self check, a sort of "does this word even make sense transcribed this way". You can certainly do this yourself as a way of validating that everything got sanely converted.

share|improve this answer
1  
Yes, a spell-check is a good idea. I've done this type of thing many times.. and there is no one-stop solution (in my experience)... I used four steps. 1. Tune the OCR; 2. sed; 3. a case-statement script for quirky 'specials'; and 4. A spell checker... –  Peter.O May 27 '11 at 8:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.