0

This is the content of a text file, and I want to only keep BMP.

这是无效的字符  𡛂
这是正常的字符  哈

In the How to remove non UTF-8 characters from text file , I find there is a solution, but it doesn't work here.

iconv -f utf-8 -t utf-8 -c test.txt

enter image description here

The utf-8 of 𡛂 is F0A19B82 (located in CJK Unified Ideographs Extension B), utf-8 of 哈 is E59388(located in CJK Unified Ideographs).

If I want to delete all characters(the size greater than 3 Bytes) like 𡛂 in text file , how to do it?

7
  • What makes these characters "non-UTF-8"? Commented May 14, 2018 at 2:07
  • @IgnacioVazquez-Abrams Because the size of 𡛂 is four Bytes, it should belongs to utf8mb4 not standard utf-8.
    – partida
    Commented May 14, 2018 at 2:12
  • Then you're asking a completely different question. You want to only keep characters in the BMP. Commented May 14, 2018 at 2:13
  • And "standard UTF-8" includes up to 7-byte characters (although none beyond 4 bytes currently exist). Commented May 14, 2018 at 2:14
  • 3
    Use of the term utf8mb4 indicates that what you really have is a mysql problem. What mysql calls utf8mb4 is what the rest of the world calls UTF-8. What mysql calls utf8 is a restricted subset of UTF-8 that has no standard name. This is why it is inaccurate to refer to your goal as "deleting non-utf8". It's more like "mangling standard UTF-8 to squeeze it into a stupid encoding that only mysql uses".
    – user41515
    Commented May 14, 2018 at 3:01

2 Answers 2

2

Pass it through a charset that doesn't support the Supplementary Multilingual Plane, such as UCS-2.

$ iconv -f utf-8 -t ucs-2 -c input.txt | iconv -f ucs-2 -t utf-8
这是无效的字符  
这是正常的字符  哈
2

Without resorting to tricks, perl is able to do exactly what you ask for:

$ perl -Mopen=locale -ne 's/[^\x{1}-\x{FFFF}]//g; print' infile
这是无效的字符
这是正常的字符  哈

Please take a look at this answer for additional solutions.

4

Not the answer you're looking for? Browse other questions tagged .