64

I have a file in UTF-8 encoding with BOM and want to remove the BOM. Are there any linux command-line tools to remove the BOM from the file?

$ file test.xml
test.xml:  XML 1.0 document, UTF-8 Unicode (with BOM) text, with very long lines
|improve this question
76

If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of sed) will remove the BOM if it exists, or make no changes if it doesn't.

sed '1s/^\xEF\xBB\xBF//' < orig.txt > new.txt

You can also overwrite the existing file with the -i option:

sed -i '1s/^\xEF\xBB\xBF//' orig.txt
|improve this answer
  • 4
    this may not work in a utf8 locale, but prepending a locale override to c or posix will always work. – hildred Jul 23 '17 at 15:29
  • 3
    @hildred I've tested it with the en_US.UTF-8 locale and it worked. When will it fail? – m13r Jul 24 '17 at 6:55
  • 2
    @m13r, It depends on the version of sed and compile options. In the failure case a very new version of sed with Unicode character classes will bring the three byte sequence in as a single character which does not match the three character sequence. However in such case you can do a sixteen bit character match. However this is a new feature and not universally present. If you want to test I recommend compiling the latest version. – hildred Jul 24 '17 at 16:25
  • 4
    To fix it to work with a unicode-enabled sed do LC_ALL=C sed '1s/^\xEF\xBB\xBF//' – Joshua Jul 24 '17 at 17:41
  • 1
    @mazunki, 1s/ means only search the first line; other lines are unaffected. The ^ means only match at the start of the (first) line. \xEF\xBB\xBF is the UTF-8 BOM (escaped hex string). // means replace with nothing. I could have added 1 to the end (for 1s/^xEF\xBB\xBF//1), which would mean only match the first occurrence of the pattern on the line. But as the the search is anchored with ^, this won't make any difference. If the file doesn't have the BOM at the start of the first line, the pattern won't match, and thus no change is made. – CSM Oct 27 at 18:47
65

A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes.

dos2unix will remove it and also take care of other idiosyncrasies of Windows text files.

dos2unix test.xml
|improve this answer
  • 17
    I agree that a UTF-8 encoded BOM does not make sense, but believe it or not, there are lots of people who think it is a great idea that helps differentiate UTF-8 from other 8-bit encodings. So it is a matter of taste. Windows Notepad adds a BOM on purpose. – Johan Myréen Jul 23 '17 at 14:02
  • 17
    What does it matter if it makes sense or not, when the context is just a question on how to remove it? According to Wikipedia, Notepad requires the BOM to recognize a file as UTF-8, and Google Docs also adds it while exporting a file as text. I doubt they all do it by mistake. – ilkkachu Jul 23 '17 at 14:09
  • Comments are not for extended discussion; this conversation has been moved to chat. – terdon♦ Jul 24 '17 at 14:07
  • 1
    Is there a way of not converting the line endings and just remove the BOM with dos2unix? – m13r Jul 25 '17 at 7:55
  • 2
    @m13r Then use the sed script in this answer. That will remove only the bom (if it exist), nothing else will be changed. – Arrow Jul 26 '17 at 5:51
27

It is possible to remove the BOM from a file with the tail command:

tail -c +4 withBOM.txt > withoutBOM.txt
|improve this answer
  • 2
    Why 4? The BOM has 3 byte. – deviantfan Jul 23 '17 at 17:12
  • 10
    @deviantfan Which is why you need to start at the 4th byte if you want to skip it. – Stéphane Chazelas Jul 23 '17 at 18:33
  • 9
    tail is using 1 based indexing?! WTF! – CodesInChaos Jul 23 '17 at 19:31
  • 5
    @CodesInChaos, tail -c -1 or tail -c 1 (what tail is generally used for) is the content starting with the last byte, tail -c +1 starting with the first byte. tail -c 0/tail -c +0 for that would be a lot more unintuitive. – Stéphane Chazelas Jul 23 '17 at 23:05
  • 2
    @deviantfan: (dd bs=1 count=3 of=/dev/null; cat) <input >output. Or with GNU (head -c3 >/dev/null; cat) -- even in UTF8 or other non-singlebyte locale; GNU head does 'char'=byte. – dave_thompson_085 Jul 24 '17 at 6:16
20

Using VIM

  1. Open file in VIM:

    vi text.xml

  2. Remove BOM encoding: 

    :set nobomb

  3. Save and quit:

    :wq
|improve this answer
  • Oddly with vim 8 on a mac, I have a csv utf-8 file made by Excel and it starts with <feff>, yet :set nobomb doesn't modify or remove it. – dlamblin Oct 9 at 21:11
5

You can use

LANG=C LC_ALL=C sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- filename

to remove the byte order mark from the beginning of the file, if it has any, as well as convert any CR LF newlines to LF only. The LANG=C LC_ALL=C tells the shell you want the command to run in the default C locale (also known as the default POSIX locale), where the three bytes forming the Byte Order Mark are treated as bytes. The -i option to sed means in-place. If you use -i.old, then sed saves the original file as filename.old, and the new file (with the modifications, if any) as filename.

I personally like to have this as ~/bin/fix-ms; for example, as

#!/bin/dash
export LANG=C LC_ALL=C
if [ $# -gt 0 ]; then
    for FILE in "$@" ; do
        sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- "$FILE" || exit 1
    done
else
    exec sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//'
fi

so that if I need to apply this to say all C source files and headers (my old code from the MS-DOS era, for example!), I just run

find . -name '*.[CHch]' -print0 | xargs -r0 ~/bin/ms-fix

or, if I just want to look at such a file, without modifying it, I can run

~/bin/ms-fix < filename | less

and not see the ugly <U+FEFF> in my UTF-8 terminal.

|improve this answer
  • Why not simply sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- "$@"? – Stéphane Chazelas Jul 24 '17 at 14:02
  • @StéphaneChazelas: Because I want the script to exit immediately if there is an issue with a replacement, which sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- "$@" does not do; it does return an exit code, but it processes all files listed in the argument list before exiting. – Nominal Animal Jul 24 '17 at 14:24
  • @StéphaneChazelas: The -- before the file name(s) is, of course, important: without it, file names beginning with a dash may be considered options by sed. I edited those into my answer; thank you for the reminder! – Nominal Animal Jul 24 '17 at 14:27
0

Recently I found this tiny command-line tool which adds or removes the BOM on arbitary UTF-8 encoded files: UTF BOM Utils (new link at github)

Little drawback, you can download only the plain C++ source code. You have to create the makefile (with CMake, for example) and compile it by yourself, binaries are not provided on this page.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.