I‘m building a script to extract some data from a website with broken character encoding:
- The html header claims it‘s iso-8859-1, but it‘s not
wget
ting the file shows me it‘s actually utf-8, but with wrong characters- Reverse engineering shows me that someone managed to use the windows codepage 1252 as unicode code!
So for example the backtick is 0x91 in codepage 1252 and is U+0091 in this page. Weird. Amazingly, web browsers seem to be able to repair this automagically.
My question: Which tool can help me clean this mess? (Not by hand! This is a dynamic website with hundreds of pages and I saw at least six different false encodings.)
iconv
, but I wouldn't have tested it on irregular encoded input. But it worked for some reason.