0

This question already has an answer here:

This is the content of a text file, and I want to only keep BMP.

这是无效的字符  𡛂
这是正常的字符  哈

In the How to remove non UTF-8 characters from text file , I find there is a solution, but it doesn't work here.

iconv -f utf-8 -t utf-8 -c test.txt

enter image description here

The utf-8 of 𡛂 is F0A19B82 (located in CJK Unified Ideographs Extension B), utf-8 of 哈 is E59388(located in CJK Unified Ideographs).

If I want to delete all characters(the size greater than 3 Bytes) like 𡛂 in text file , how to do it?

marked as duplicate by Isaac, shirish, agc, Kiwy, G-Man May 16 '18 at 3:16

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • What makes these characters "non-UTF-8"? – Ignacio Vazquez-Abrams May 14 '18 at 2:07
  • @IgnacioVazquez-Abrams Because the size of 𡛂 is four Bytes, it should belongs to utf8mb4 not standard utf-8. – partida May 14 '18 at 2:12
  • Then you're asking a completely different question. You want to only keep characters in the BMP. – Ignacio Vazquez-Abrams May 14 '18 at 2:13
  • And "standard UTF-8" includes up to 7-byte characters (although none beyond 4 bytes currently exist). – Ignacio Vazquez-Abrams May 14 '18 at 2:14
  • 3
    Use of the term utf8mb4 indicates that what you really have is a mysql problem. What mysql calls utf8mb4 is what the rest of the world calls UTF-8. What mysql calls utf8 is a restricted subset of UTF-8 that has no standard name. This is why it is inaccurate to refer to your goal as "deleting non-utf8". It's more like "mangling standard UTF-8 to squeeze it into a stupid encoding that only mysql uses". – user41515 May 14 '18 at 3:01
2

Pass it through a charset that doesn't support the Supplementary Multilingual Plane, such as UCS-2.

$ iconv -f utf-8 -t ucs-2 -c input.txt | iconv -f ucs-2 -t utf-8
这是无效的字符  
这是正常的字符  哈
2

Without resorting to tricks, perl is able to do exactly what you ask for:

$ perl -Mopen=locale -ne 's/[^\x{1}-\x{FFFF}]//g; print' infile
这是无效的字符
这是正常的字符  哈

Please take a look at this answer for additional solutions.

Not the answer you're looking for? Browse other questions tagged or ask your own question.