I have a set of tex-files with mixed encodings, e.g. (subset of output of file -i *.tex)

f1.tex: text/plain; charset=utf-8
f2.tex: text/plain; charset=utf-8
f3.tex: text/x-tex; charset=us-ascii
f4.tex: text/plain; charset=iso-8859-1
f5.tex: text/plain; charset=us-ascii

I want to convert them all to utf-8, especially those encoded iso-8859-1. I can do this manually using (or similar)

iconv -f ISO-8859-1 -t UTF-8 f4.tex > tmp && mv tmp f4.tex

but I thought this would be nicely possible using a combination of awk with the above, i.e. running file -i parsing this for file/encoding combinations using awk and performing the encoding conversion accordingly using iconv.

My knowledge of awk is rather limited. I got no further than this:

$ file -i *.tex | awk '{print $1, $3}'
f1.tex: charset=utf-8
f2.tex: charset=utf-8
f3.tex: charset=us-ascii
f4.tex: charset=iso-8859-1
f5.tex: charset=us-ascii

Any help appreciated! Especially, I don't know how I can strip the colon : and the charset= substrings off the columns.


Seems it's much better to use sed here insted of awk:

file -i *tex | sed \ 
's/^\([^:]*\): .*set=\(.*\)/iconv -f \2 -t UTF-8 \1 > tmp \&\& mv tmp \1/e'

It would execute commands that sed will make based on file -i output. If you want to look at commands list without executing just remove e flag at the end of sed script like this:

file -i *tex | sed \
's/^\([^:]*\): .*set=\(.*\)/iconv -f \2 -t UTF-8 \1 > tmp \&\& mv tmp \1/'
  • I'm intrigued by the e flag, but it probably is a GNU extension? (That is, FreeBSD's sed doesn't know it.) – sr_ Apr 18 '12 at 6:57
  • Yes, it is a GNU extension. I forgot to write about it. To use smth like that in non-GNU environment you can use contruction echo "ehco Hello" | sed 's/hc/ch/' | bash . – rush Apr 18 '12 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.