I have a file in UTF-8 encoding with BOM and want to remove the BOM. Are there any linux command-line tools to remove the BOM from the file?
$ file test.xml test.xml: XML 1.0 document, UTF-8 Unicode (with BOM) text, with very long lines
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
If you're not sure if the file contains a UTF-8 BOM, then this (assuming the GNU implementation of
sed) will remove the BOM if it exists, or make no changes if it doesn't.
sed '1s/^\xEF\xBB\xBF//' < orig.txt > new.txt
You can also overwrite the existing file with the
sed -i '1s/^\xEF\xBB\xBF//' orig.txt
If you are using the BSD version of
sed (eg macOS) then you need to have bash do the escaping:
sed $'1s/\xef\xbb\xbf//' < orig.txt > new.txt
You can use
LANG=C LC_ALL=C sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- filename
to remove the byte order mark from the beginning of the file, if it has any, as well as convert any CR LF newlines to LF only. The
LANG=C LC_ALL=C tells the shell you want the command to run in the default C locale (also known as the default POSIX locale), where the three bytes forming the Byte Order Mark are treated as bytes. The
-i option to sed means in-place. If you use
-i.old, then sed saves the original file as
filename.old, and the new file (with the modifications, if any) as
I personally like to have this as
~/bin/fix-ms; for example, as
#!/bin/dash export LANG=C LC_ALL=C if [ $# -gt 0 ]; then for FILE in "$@" ; do sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' -i -- "$FILE" || exit 1 done else exec sed -e 's/\r$// ; 1 s/^\xef\xbb\xbf//' fi
so that if I need to apply this to say all C source files and headers (my old code from the MS-DOS era, for example!), I just run
find . -name '*.[CHch]' -print0 | xargs -r0 ~/bin/ms-fix
or, if I just want to look at such a file, without modifying it, I can run
~/bin/ms-fix < filename | less
and not see the ugly
<U+FEFF> in my UTF-8 terminal.
Little drawback, you can download only the plain C++ source code. You have to create the makefile (with CMake, for example) and compile it by yourself, binaries are not provided on this page.