Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a bash script which enumerates through every *.php file in a directory and applies iconv to it. This gets output in STDOUT.

Since adding the -o parameter ( in my experience ) actually writes a blank file probably before the conversion takes place, how can I adjust my script so it does the conversion, then overwrites the input file?

for file in *.php
do
    iconv -f cp1251 -t utf8 "$file"
done
share|improve this question
add comment

2 Answers

up vote 27 down vote accepted

This isn't working because iconv first creates the output file (since the file already exists, it truncates it), then starts reading its input file (which is now empty). Most programs behave this way.

Create a new, temporary file for the output, then move it into place.

for file in *.php
do
    iconv -f cp1251 -t utf8 -o "$file.new" "$file" &&
    mv -f "$file.new" "$file"
done

Colin Watson's sponge utility (included in Joey Hess's moreutils) automates this:

for file in *.php
do
    iconv -f cp1251 -t utf8 "$file" | sponge "$file"
done

This answer applies not just to iconv but to any filter program. A few special cases are worth mentioning:

  • GNU sed and Perl -p have a -i option to replace files in place.
  • If your file is extremely large, your filter is only modifying or removing some parts but never adding things (e.g. grep, tr, sed 's/long input text/shorter text/'), and you like living dangerously, you may want to genuinely modify the file in place (the other solutions mentioned here create a new output file and move it into place at the end, so the original data is unchanged if the command is interrupted for any reason).
share|improve this answer
1  
I'm not quite sure whether the authorship of sponge should be attributed exclusively to Joey Hess; it's the package moreutils that includes sponge that he maintains, but as regards the origin of sponge, by following the links from the homepage of moreutils, I've found it originally posted and suggested for inclusion by Colin Watson: "Joey writes about the lack of new tools that fit into the Unix philosophy. My favourite of such things I've written is sponge" (Mon, 06 Feb 2006). –  imz -- Ivan Zakharyaschev Mar 30 '11 at 14:13
    
+1 for information on sponge –  enzotib Nov 24 '11 at 8:13
add comment

An alternative is recode, which uses the libiconv library for some conversions. Its behavior is to replace the input file with the output, so this will work:

for file in *.php
do
    recode cp1251..utf8 "$file"
done

As recode accepts multiple input files as parameter, you can spare the for loop:

recode cp1251..utf8 *.php
share|improve this answer
1  
Thanks, this deserves more upvotes. Just wondering where is stared in manual about the 2 dots between the encodings... –  neurino Nov 20 '12 at 21:14
    
“REQUEST often looks like BEFORE..AFTER, with BEFORE and AFTER being charsets.” That manual is indeed hard to follow with all those double dots (which are part of the syntax) and triple dots (which mean more of this). An advice: try info recode instead. Is more verbose. –  manatwork Nov 21 '12 at 6:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.