1

I copied text from a PDF file in bash - vi and I get lines like this:

echo <93>The Hypotenuse is $hypotenuse<94>

I tried to change that in vi by:

s/<93>/"/g

But:

E486: Pattern not found: <93>

And by sed:

sed 's/'`echo "\093"`'/"/g' par.sh

And:

sed 's/'`echo "\094"`'/"/g' par.sh

And:

sed 's/\<93\>/\"/g' aa.sh
echo В“The Hypotenuse is $hypotenuseВ”

Could you tell me what I should use?

2
  • Your vi approach should have worked. If it didn't, you have non-printing characters. Could you show us the actual file?
    – terdon
    Jun 24, 2015 at 13:04
  • If you are using a terminal emulator that supports colors, make sure that the <93> stuffs are not colored in vi. If they are, they must be hex-encoded unprintable characters. And your approach using echo is badly wrong -- You should use -e for backslash escapes, and \0oo is actually octal. The final resulting command would be sed -i.bak -e s/[$(echo -e '\x93')$(echo -e '\x94')]/\"/g par.sh. Jun 24, 2015 at 13:13

4 Answers 4

3

In vi you need to enter the key sequence for each of the 2 special characters, not the ascii chars for their display (if you place the cursor on them you'll see the entire ascii sequence is highlighted):

  • Ctrl+v, x, 9, 3 instead of <93>
  • Ctrl+v, x, 9, 4 instead of <94>

So the search cmd would look something like this:

:%s /Ctrl+v, x, 9, 3/"/g

0
1

With sed, you can use literal characters:

sed -e 's/”/"/g' -e 's/“/"/g' file

or using perl with Unicode code point:

perl -CSD -pe 's/\x{201C}|\x{201D}/"/g' file
3
  • $ sed -e 's/”/"/g' -e 's/“/"/g' aa.sh > a.sh $ more a.sh echo В"The Hypotenuse is $hypotenuseВ"
    – avsun
    Jun 24, 2015 at 13:24
  • @avsun: So does it work for you?
    – cuonglm
    Jun 24, 2015 at 13:29
  • I have success with it only in vi, by sed I have an issue because <93> in sed looks like B”
    – avsun
    Jun 25, 2015 at 11:32
0

This is another way but using perl.

perl -i.bak -pe 's/[^[:ascii:]]//g' file
  • The -i flag creates a backup of the original file.
0

Found a similar question in StackOverflow, this solution worked:

iconv -f cp1252 -t utf8 inputfile.csv > outputfile.csv

Short explanation- the characters are coming in cp1252 encoding, you can convert it to utf8.

1
  • Please consider adding a link to the SO question/answer, to help the OP and other readers fully understand.
    – 0xSheepdog
    Jul 4, 2019 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.