1

I copied text from a PDF file in bash - vi and I get lines like this:

echo <93>The Hypotenuse is $hypotenuse<94>

I tried to change that in vi by:

s/<93>/"/g

But:

E486: Pattern not found: <93>

And by sed:

sed 's/'`echo "\093"`'/"/g' par.sh

And:

sed 's/'`echo "\094"`'/"/g' par.sh

And:

sed 's/\<93\>/\"/g' aa.sh
echo В“The Hypotenuse is $hypotenuseВ”

Could you tell me what I should use?

2
  • Your vi approach should have worked. If it didn't, you have non-printing characters. Could you show us the actual file?
    – terdon
    Jun 24, 2015 at 13:04
  • If you are using a terminal emulator that supports colors, make sure that the <93> stuffs are not colored in vi. If they are, they must be hex-encoded unprintable characters. And your approach using echo is badly wrong -- You should use -e for backslash escapes, and \0oo is actually octal. The final resulting command would be sed -i.bak -e s/[$(echo -e '\x93')$(echo -e '\x94')]/\"/g par.sh. Jun 24, 2015 at 13:13

4 Answers 4

3

In vi you need to enter the key sequence for each of the 2 special characters, not the ascii chars for their display (if you place the cursor on them you'll see the entire ascii sequence is highlighted):

  • Ctrl+v, x, 9, 3 instead of <93>
  • Ctrl+v, x, 9, 4 instead of <94>

So the search cmd would look something like this:

:%s /Ctrl+v, x, 9, 3/"/g

0
1

With sed, you can use literal characters:

sed -e 's/”/"/g' -e 's/“/"/g' file

or using perl with Unicode code point:

perl -CSD -pe 's/\x{201C}|\x{201D}/"/g' file
3
  • $ sed -e 's/”/"/g' -e 's/“/"/g' aa.sh > a.sh $ more a.sh echo В"The Hypotenuse is $hypotenuseВ"
    – avsun
    Jun 24, 2015 at 13:24
  • @avsun: So does it work for you?
    – cuonglm
    Jun 24, 2015 at 13:29
  • I have success with it only in vi, by sed I have an issue because <93> in sed looks like B”
    – avsun
    Jun 25, 2015 at 11:32
0

This is another way but using perl.

perl -i.bak -pe 's/[^[:ascii:]]//g' file
  • The -i flag creates a backup of the original file.
0

Found a similar question in StackOverflow, this solution worked:

iconv -f cp1252 -t utf8 inputfile.csv > outputfile.csv

Short explanation- the characters are coming in cp1252 encoding, you can convert it to utf8.

1
  • Please consider adding a link to the SO question/answer, to help the OP and other readers fully understand.
    – 0xSheepdog
    Jul 4, 2019 at 17:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .