Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a set of tex-files with mixed encodings, e.g. (subset of output of file -i *.tex)

f1.tex: text/plain; charset=utf-8
f2.tex: text/plain; charset=utf-8
f3.tex: text/x-tex; charset=us-ascii
f4.tex: text/plain; charset=iso-8859-1
f5.tex: text/plain; charset=us-ascii

I want to convert them all to utf-8, especially those encoded iso-8859-1. I can do this manually using (or similar)

iconv -f ISO-8859-1 -t UTF-8 f4.tex > tmp && mv tmp f4.tex

but I thought this would be nicely possible using a combination of awk with the above, i.e. running file -i parsing this for file/encoding combinations using awk and performing the encoding conversion accordingly using iconv.

My knowledge of awk is rather limited. I got no further than this:

$ file -i *.tex | awk '{print $1, $3}'
f1.tex: charset=utf-8
f2.tex: charset=utf-8
f3.tex: charset=us-ascii
f4.tex: charset=iso-8859-1
f5.tex: charset=us-ascii

Any help appreciated! Especially, I don't know how I can strip the colon : and the charset= substrings off the columns.

share|improve this question
up vote 2 down vote accepted

Seems it's much better to use sed here insted of awk:

file -i *tex | sed \ 
's/^\([^:]*\): .*set=\(.*\)/iconv -f \2 -t UTF-8 \1 > tmp \&\& mv tmp \1/e'

It would execute commands that sed will make based on file -i output. If you want to look at commands list without executing just remove e flag at the end of sed script like this:

file -i *tex | sed \
's/^\([^:]*\): .*set=\(.*\)/iconv -f \2 -t UTF-8 \1 > tmp \&\& mv tmp \1/'
share|improve this answer
I'm intrigued by the e flag, but it probably is a GNU extension? (That is, FreeBSD's sed doesn't know it.) – sr_ Apr 18 '12 at 6:57
Yes, it is a GNU extension. I forgot to write about it. To use smth like that in non-GNU environment you can use contruction echo "ehco Hello" | sed 's/hc/ch/' | bash . – rush Apr 18 '12 at 8:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.