I have a Java class which the compiler refuses to compile due to \ufeff at the start of the file. I can view the fact that the BOM is present by vim -b file.java, but neither xxd nor hexdump show the two bytes. Is there some way to make them do so?
2
3
The U+FEFF character is encoded in UTF-8 over 3 bytes: ef bb bf.
xxd or hexdump shows you the byte content, so those 3 bytes, not the character that those 3 bytes encode like vim -b does.
To remove that BOM (which doesn't make sense in UTF-8) and fix other idiosyncrasies of Microsoft text files (which is likely the source of your problem), you can use dos2unix.
$ printf '\ufefffoobar\r\n' | hd
00000000 ef bb bf 66 6f 6f 62 61 72 0d 0a |...foobar..|
0000000b
$ printf '\ufefffoobar\r\n' | uconv -x name
\N{ZERO WIDTH NO-BREAK SPACE}\N{LATIN SMALL LETTER F}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER O}\N{LATIN SMALL LETTER B}\N{LATIN SMALL LETTER A}\N{LATIN SMALL LETTER R}\N{<control-000D>}\N{<control-000A>}
$ printf '\ufefffoobar\r\n' | uconv -x hex
\uFEFF\u0066\u006F\u006F\u0062\u0061\u0072\u000D\u000A
$ printf '\ufefffoobar\r\n' | dos2unix | hd
00000000 66 6f 6f 62 61 72 0a |foobar.|
00000007
-
Silly me, I was looking for the bytes
fe ff, but now I realise that that's the codepoint, not the utf8 code. – oarfish Apr 25 '17 at 14:48
