I need to search and replace all occurences of an unknown character in some files having the same name.

Opening such files with vi, I read <91> code for that character. Opening them with nano, I read a "question mark" in a diamond (black rhumble).

I would like to replace such unknown character with a quote ('). I'm trying many ways with no luck.

I tried:

find ./ -name filename.txt -exec perl -i~ -pe "s/\x91/'/" {} \;

find ./ -name filename.txt -exec sed -i "s/\x91/'/g" {} \;

EDIT More info on the character:

Hexadecimal: 91 68 74 74
Decimal: 145 104 116 116
Octal: 221 150 164 164
Binary: 10010001 01101000 01110100 01110100

LC_ALL=C sed -n l < file


If you need more, ask!

  • In which way does sed -i "s/\x91/'/g" on that file not work? Commented Feb 21, 2014 at 9:42

2 Answers 2


You should have a look using hexdump -C and find the bytes around it. Presuming UTF-8, what vi shows as <91> (decimal 145, a unicode point meaningless in text) would be two bytes, 0xc2 and 0x91.

It's implied your substitutions didn't work at all, but if what you did was just replace 0x91 with 0x27, you'll have invalidated the UTF-8 (the second byte of a two byte sequence always has the high bit set, i.e. is >= 0x80). This might complicate your analysis, although vi should then show it as ?'.

That said, I tested this and it works:

use strict;
use warnings FATAL => qw(all);

my $data = "";
my $file = $ARGV[0];

while (<>) {
    $data .= $_;

open my $out, '>', $file || die "Could not write $file.";
print $out $data;
close $out;  

If $ARGV[0] exists when <> is referenced, perl pops this off the argument stack and takes it as a filepath to use for input (I find short scripts easier to tweak and work with than one liners, BTW). This accumulates in memory (fine as long as the files are not massive), whereas perl -i renames the original file to avoid edit-in-place race conditions (see perldoc perlrun).

So you could use that:

  find . -name "*.txt" -exec whatever.pl {} +
  • it doesn't work, the question mark remains...
    – jasmines
    Commented Feb 21, 2014 at 8:10
  • Did you check it in hexdump -C to see what's actually there?
    – goldilocks
    Commented Feb 21, 2014 at 13:11

If it is indeed the character U+0091 (0xc2 0x91 in UTF-8 encoding) and not the byte 0x91, then:

PERLIO=:utf8 perl -pi -e "s/\N{U+0091}/'/g" file

Would convert it to '.

With GNU sed:

sed -i "s/\xc2\x91/'/" file


However, in your case, the file is not in UTF-8. UTF-8 characters are one byte, only for ASCII characters (for values 0 to 0x7F). The other characters are represented by two or more bytes whose value is greater than 0x7F. So a 0x91 byte, with no byte greater than 0x7F around it cannot be found in a utf-8 file.

More likely, your file is in a single-byte character set, most likely some Microsoft one like windows-1252.

In windows-1252, 0x91 is the left single quote character. The unicode equivalent is U+2018 which in UTF-8 is written 0xe2 0x80 0x98.

If you want to convert your file to UTF-8, best is probably to use a dedicated tool for that. Like:

recode windows-1252..utf8 < file


iconv -f windows-1252 -t utf-8 < file

Or if you want to do it for every filename.txt:

find . -type f -name filename.txt -exec sh -Cc '
  for file do
    mv "$file" "$file~" &&
      iconv -f windows-1252 -t utf-8 < "$file~" > "$file"
  done' sh {} +
  • it doesn't work, the question mark remains...
    – jasmines
    Commented Feb 21, 2014 at 8:10
  • @jasmines Then it's not a U+0091. Please add the output of LC_ALL=C sed -n l < file to the question. Commented Feb 21, 2014 at 8:30
  • it seems to be \221
    – jasmines
    Commented Feb 21, 2014 at 8:33
  • I can't convert because is not a single file... I need to batch and recursively search and replace.
    – jasmines
    Commented Feb 21, 2014 at 8:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .