6

I have two files encoded in UTF-8 with/without BOM:

/tmp/bom$ ls
list.bom.txt  list.nobom.txt
/tmp/bom$ cat list.nobom.txt 
apple
banana
avocado
寿司
melon
/tmp/bom$ diff list.nobom.txt list.bom.txt 
1c1
< apple
---
> apple
/tmp/bom$ file list.nobom.txt list.bom.txt 
list.nobom.txt: UTF-8 Unicode text
list.bom.txt:   UTF-8 Unicode (with BOM) text

The only diff between two files is header BOM EF BB BF.

Then, in order to filter the lines that begin with 'a', I write a short awk script using a caret.

/tmp/bom$ gawk '/^a.*/' list.nobom.txt
apple
avocado
/tmp/bom$ gawk '/^a.*/' list.bom.txt
avocado

Unfortunately, with header BOM, apple in the first line is ignored.

Therefore, my question is: Is there any way to handle this?

I consider three solutions:

  1. Write BOM bytes directly. For example,

    gawk 'BEGIN { pat = "^(\xef\xbb\xbf)?a.*" } $0 ~ pat { print }'
    

    works in UTF-8. However, this doesn't handle other encodings. Moreover, if there is U+FEFF used as Zero Width Non-Breaking Space (see comments), the above script fails in some cases.

  2. Delete BOM bytes by re-encoding with nkf. For example,

    nkf --oc=UTF-8 list.bom.txt | gawk '/^a.*/'
    

    works. However, I wonder if there is a more sophisticated way.

  3. [ADDED] This is an improvement of the first one, using bash feature.

    gawk -v bom="$(echo -e '\uFEFF')" '
        NR == 1 {
            pat = "^" bom;
            sub(pat, "")
        }
        /^a.*/ {
            print
        }
    '
    

    This works for both UTF-8 with/without BOM. However this doesn't works for UTF-16 in my environment. So, the second solution is better.

Moreover, I think this is also the problem for grep, sed, or other scripts using regular expression matching. So, if there is a general solution, it would be more appreciated.

  • The file with the BOM does not start with 'a', it starts with the BOM (also known as Unicode character Zero Width No-Break Space). The files do not have the same content, just as a file with starting with a regular space character differs from a file starting with the character 'a'. – Johan Myréen Jan 20 '17 at 11:57
  • @JohanMyréen Thanks for the comment! However, U+FEFF (BOM, or Zero Width No-Break Space) is now neither control nor graphic character, according to Unicode Standard, Version 9.0.0, Section 23.8 "Specials". This section also says "(Except for compatibility,) U+FEFF is not used with the semantics of zero width no-break space." And I think this time it's not a compatibility matter. (Is this correct?) – nekketsuuu Jan 20 '17 at 12:30
  • Anyway, I know backward compatibility is very important, so it's OK that the beginning 'a' of the file with BOM is not matched in the intuitive way. But I want to know an option or something for this problem, if it exists. – nekketsuuu Jan 20 '17 at 12:30
  • 1
    Yes, I am aware that the definition changed and U+FEFF is now only used for BOM. I find it unfortunate that the BOM has found its way to UTF-8, where it is not needed for its original purpose (because UTF-8 does not have a byte order). Using BOM for UTF-8 files is just asking for trouble, which this question has shown. My recommendation is to not use BOM for UTF-8 at all. You can't use it to decide if a file is encoded using UTF-8 or not, because there are UTF-8 files without BOM anyway. The Unicode standard recommends to not use BOM for UTF-8 files. – Johan Myréen Jan 20 '17 at 14:39
  • I think so too.... (But then why Visual Studio use utf-8 with BOM by default?) So my second solution (converting all to uft-8 without BOM) is recommended? – nekketsuuu Jan 20 '17 at 16:20
3

A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes.

dos2unix will remove it and also take care of other idiosyncrasies of Windows text files.

dos2unix < file.win.txt | awk ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.