1
vote
1answer
28 views

What does “position” mean in iconv error message?

I get error when converting a large file: $ iconv -f GB2312 -t UTF-8 2001.txt -o 2001_u.txt iconv: illegal input sequence at position 245256667 What does the position mean in this error? I tried it ...
2
votes
3answers
81 views

How to convert html entities to readable text?

I want html number entities like ę and want to convert it to real character. I have emails mostly from linkedin that look like this: chciałabym zapytać, czy ...
0
votes
2answers
1k views

iconv illegal input sequence- why?

While trying to convert a text file into its ASCII equivalent, I get error message that iconv: illegal input sequence at position. Command I use is iconv -f UTF-8 -t ascii//TRANSLIT file The ...
3
votes
2answers
374 views

tr complains of “Illegal byte sequence”

I'm brand new to UNIX and I am using Kirk McElhearn's "The Mac OS X Command Line" to teach myself some commands. I am attempting to use tr and grep so that I can search for text strings in a regular ...
3
votes
1answer
302 views

Process a file that starts with a BOM (FF FE)

I received a .csv file with the FF FE BOM: $ head -n1 dotan.csv | hd 00000000 ff fe 41 00 64 00 20 00 67 00 72 00 6f 00 75 00 |..A.d. .g.r.o.u.| When using awk to parse it I'm getting a bunch of ...
1
vote
1answer
70 views

How to export string about France to Microsoft excel file?

One string file Str.rc about French language is here: ID_STR_BRIGHTNESS;,"Luminosité" ID_STR_CHILE_EASTER_ISLAND;,"Île de Pâques" ID_STR_CURRENT_CH;,"Saisie chaîne" ID_STR_DETAILS;,"Détails" ...... ...
5
votes
2answers
685 views

Lowercase all but the first (Uppercase) letter from UPPERCASE in cyrillic

To make all letters a lower case except the first letter. The first letter would look like "Uppercase" after I changed (from UPPERCASE in cyrillic). The rest (not UPPERCASED) leave unchanged. I'm ...
6
votes
3answers
600 views

How to replace all percent-encoded UTF-8 substrings with plain UTF-8 text?

I've got a html file with a lot of %-encoded UTF-8 text in URLs. For example "%D1%80%D0%B5%D1%81%D1%83%D1%80%D1%81%D1%8B" stands for "ресурсы" ("resources" in Russian). The task is to replace all ...
2
votes
2answers
437 views

Remove copyright symbol

I'm trying to parse a RSS feed on the command line. The code works so far but the feed contains a copyright symbol which I try to remove (it is latin1 encoded). How do I remove the copyright symbol ...
4
votes
4answers
3k views

How to convert hex chars to normal chars?

I tried, but I'm stuck at "escaping" the "sed's": sed -i 's/\\x0/NUL/g' $1 sed -i 's/\\x1/SOH/g' $1 sed -i 's/\\x2/STX/g' $1 sed -i 's/\\x3/ETX/g' $1 sed -i 's/\\x4/EOT/g' $1 sed -i 's/\\x5/ENQ/g' ...
2
votes
1answer
203 views

XML text file with `^@` characters in it?

I have an XML file that I need to parse. When I open it in nano, nano give me the message (converted from Mac format). However between each character, there is a ^@ sequence, like so: ^@t^@h^@e^@ ...
6
votes
2answers
150 views

What is the connection between a gedit bug and a Unix-&-Linux Q/A href?

While answering a Unix-&-Linux question, I observed that Gedit and two other editors, Leafpad and Medit (I tested 12 editors altogether) exhibit the a certain bug. As it turns out, the bug is ...
8
votes
2answers
13k views

How can I test the encoding of a text file… Is it valid, and what is it?

I have several .htm files which open in Gedit without any warning/error, but when I open these same file in Jedit, it warns me of invalid UTF-8 encoding... The html meta tag states ...
14
votes
2answers
9k views

Filtering invalid utf8

I have a text file in an unknown or mixed encoding. I want to see the lines that contain a byte sequence that is not valid UTF-8 (by piping the text file into some program). Equivalently, I want to ...