Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to search and replace all occurences of an unknown character in some files having the same name.

Opening such files with vi, I read <91> code for that character. Opening them with nano, I read a "question mark" in a diamond (black rhumble).

I would like to replace such unknown character with a quote ('). I'm trying many ways with no luck.

I tried:

find ./ -name filename.txt -exec perl -i~ -pe "s/\x91/'/" {} \;



find ./ -name filename.txt -exec sed -i "s/\x91/'/g" {} \;

EDIT More info on the character:

Hexadecimal: 91 68 74 74
Decimal: 145 104 116 116
Octal: 221 150 164 164
Binary: 10010001 01101000 01110100 01110100

LC_ALL=C sed -n l < file

\221

If you need more, ask!

share|improve this question
    
In which way does sed -i "s/\x91/'/g" on that file not work? –  Stéphane Chazelas Feb 21 at 9:42
add comment

2 Answers 2

You should have a look using hexdump -C and find the bytes around it. Presuming UTF-8, what vi shows as <91> (decimal 145, a unicode point meaningless in text) would be two bytes, 0xc2 and 0x91.

It's implied your substitutions didn't work at all, but if what you did was just replace 0x91 with 0x27, you'll have invalidated the UTF-8 (the second byte of a two byte sequence always has the high bit set, i.e. is >= 0x80). This might complicate your analysis, although vi should then show it as ?'.

That said, I tested this and it works:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

my $data = "";
my $file = $ARGV[0];

while (<>) {
    s/\xc2\x91/'/g;
    $data .= $_;
}

open my $out, '>', $file || die "Could not write $file.";
print $out $data;
close $out;  

If $ARGV[0] exists when <> is referenced, perl pops this off the argument stack and takes it as a filepath to use for input (I find short scripts easier to tweak and work with than one liners, BTW). This accumulates in memory (fine as long as the files are not massive), whereas perl -i renames the original file to avoid edit-in-place race conditions (see perldoc perlrun).

So you could use that:

  find . -name "*.txt" -exec whatever.pl {} +
share|improve this answer
    
it doesn't work, the question mark remains... –  jasmines Feb 21 at 8:10
    
Did you check it in hexdump -C to see what's actually there? –  goldilocks Feb 21 at 13:11
add comment

If it is indeed the character U+0091 (0xc2 0x91 in UTF-8 encoding) and not the byte 0x91, then:

PERLIO=:utf8 perl -pi -e "s/\N{U+0091}/'/g" file

Would convert it to '.

With GNU sed:

sed -i "s/\xc2\x91/'/" file

Edit:

However, in your case, the file is not in UTF-8. UTF-8 characters are one byte, only for ASCII characters (for values 0 to 0x7F). The other characters are represented by two or more bytes whose value is greater than 0x7F. So a 0x91 byte, with no byte greater than 0x7F around it cannot be found in a utf-8 file.

More likely, your file is in a single-byte character set, most likely some Microsoft one like windows-1252.

In windows-1252, 0x91 is the left single quote character. The unicode equivalent is U+2018 which in UTF-8 is written 0xe2 0x80 0x98.

If you want to convert your file to UTF-8, best is probably to use a dedicated tool for that. Like:

recode windows-1252..utf8 < file

Or:

iconv -f windows-1252 -t utf-8 < file

Or if you want to do it for every filename.txt:

find . -type f -name filename.txt -exec sh -Cc '
  for file do
    mv "$file" "$file~" &&
      iconv -f windows-1252 -t utf-8 < "$file~" > "$file"
  done' sh {} +
share|improve this answer
    
it doesn't work, the question mark remains... –  jasmines Feb 21 at 8:10
    
@jasmines Then it's not a U+0091. Please add the output of LC_ALL=C sed -n l < file to the question. –  Stéphane Chazelas Feb 21 at 8:30
    
it seems to be \221 –  jasmines Feb 21 at 8:33
    
I can't convert because is not a single file... I need to batch and recursively search and replace. –  jasmines Feb 21 at 8:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.