How can I delete <93> <94> symbols in file? It is equivalent " "

Question

I copied text from a PDF file in bash - vi and I get lines like this:

echo <93>The Hypotenuse is $hypotenuse<94>

I tried to change that in vi by:

s/<93>/"/g

But:

E486: Pattern not found: <93>

And by sed:

sed 's/'`echo "\093"`'/"/g' par.sh

And:

sed 's/'`echo "\094"`'/"/g' par.sh

And:

sed 's/\<93\>/\"/g' aa.sh
echo В“The Hypotenuse is $hypotenuseВ”

Could you tell me what I should use?

Answer 1

In vi you need to enter the key sequence for each of the 2 special characters, not the ascii chars for their display (if you place the cursor on them you'll see the entire ascii sequence is highlighted):

• Ctrl+v, x, 9, 3 instead of <93>
• Ctrl+v, x, 9, 4 instead of <94>

So the search cmd would look something like this:

:%s /Ctrl+v, x, 9, 3/"/g

Answer 2

With sed, you can use literal characters:

sed -e 's/”/"/g' -e 's/“/"/g' file

or using perl with Unicode code point:

perl -CSD -pe 's/\x{201C}|\x{201D}/"/g' file

Answer 3

This is another way but using perl.

perl -i.bak -pe 's/[^[:ascii:]]//g' file

• The -i flag creates a backup of the original file.

Answer 4

Found a similar question in StackOverflow, this solution worked:

iconv -f cp1252 -t utf8 inputfile.csv > outputfile.csv

Short explanation- the characters are coming in cp1252 encoding, you can convert it to utf8.