I have a file in UTF-8 encoding with BOM and want to remove the BOM. Are there any linux command-line tools to remove the BOM from the file?
$ file test.xml test.xml: XML 1.0 document, UTF-8 Unicode (with BOM) text, with very long lines
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of
sed) will remove the BOM if it exists, or make no changes if it doesn't.
sed '1s/^\xEF\xBB\xBF//' < orig.txt > new.txt
You can also overwrite the existing file with the
sed -i '1s/^\xEF\xBB\xBF//' orig.txt
A BOM doesn't make sense in UTF-8. Those are generally added by mistake by bogus software on Microsoft OSes.
dos2unix will remove it and also take care of other idiosyncrasies of Windows text files.
It is possible to remove the BOM from a file with the
tail -c +4 withBOM.txt > withoutBOM.txt
Open file in VIM:
Remove BOM encoding:
Save and quit:
You can use
LANG=C LC_ALL=C sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- filename
to remove the byte order mark from the beginning of the file, if it has any, as well as convert any CR LF newlines to LF only. The
LANG=C LC_ALL=C tells the shell you want the command to run in the default C locale (also known as the default POSIX locale), where the three bytes forming the Byte Order Mark are treated as bytes. The
-i option to sed means in-place. If you use
-i.old, then sed saves the original file as
filename.old, and the new file (with the modifications, if any) as
I personally like to have this as
~/bin/fix-ms; for example, as
#!/bin/dash export LANG=C LC_ALL=C if [ $# -gt 0 ]; then for FILE in "$@" ; do sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- "$FILE" || exit 1 done else exec sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' fi
so that if I need to apply this to say all C source files and headers (my old code from the MS-DOS era, for example!), I just run
find . -name '*.[CHch]' -print0 | xargs -r0 ~/bin/ms-fix
or, if I just want to look at such a file, without modifying it, I can run
~/bin/ms-fix < filename | less
and not see the ugly
<U+FEFF> in my UTF-8 terminal.
Little drawback, you can download only the plain C++ source code. You have to create the makefile (with CMake, for example) and compile it by yourself, binaries are not provided on this page.