I'm using Ubuntu 11.04 and wrote a small script that searches within text files for certain "tokens" and replaces with some a prewritten snippet from a template file of the same name.

The text files that are being searched will have two and only two instances of each token. The first is a plain text and the second is an html version, and there are separate snippets for each.

Here's the script:

for f in `ls -1 .templates/template_text`;
    g=`cat .templates/template_text/$f`
    find to_process/ -type f | xargs perl -i.old -p -e "s/$f/$g/";

for f in `ls -1 .templates/template_html`;
    g=`cat .templates/template_html/$f`
    find to_process/ -type f | xargs perl -i.old -p -e "s/$f/$g/g";

I'm running into a problem where even when I don't have "global" specified in the first regex, it still replaces both tokens. I'm not sure if this is because of how I'm calling perl, a bug, or some other issue.

Any help would be appreciated.

UPDATE: I was able to get the script working by using sed instead of perl.

for f in `ls -1 .templates/template_text`;
    g=`cat .templates/template_text/$f`
    h=`cat .templates/template_html/$f`
    find to_process/ -type f -print0 | xargs -0 -I {} sed -i -e "0,/$f/s/$f/$g/" -e "0,/$f/s/$f/$h/" {}

Still interested in how to get it working with the perl command though.


That's because perl reads the text file one line at a time, and applies the substitution pattern to each line -- so if there are multiple occurrences of the token in different lines, they will all get replaced.

To replace just the first occurrence in the file, you can add the -0 option, which sets the input record separator to a null character and makes perl read the whole file before making the substitution.

| improve this answer | |

s/$f/$g/ replaces the first occurrence of $f by $g on each line. If you want to replace only the first occurrence of $f in the whole file, you need to say so. This is what you eventually did in sed with 0,/$f/ s/$f/$g/ (replace $f by $g up to and including the first occurrence of $f). In Perl, you can write this in a more verbose but easier to understand way like this (note: see below for quoting issues):

perl -i -pe 'if ($n==0) {s/$f/$g/; $n=1;} elsif ($n==1) {s/$f/$h/; $n=2}'

Your code suffers from numerous quoting issues; you'll run into trouble if your file names contain whitespace, globbing characters, or unprintable characters (such as byte sequences that don't exist in the current locale). Fortunately these issues are easy to fix.

First, some generic shell issues. Always double-quote variable substitutions "$foo" and command substitutions "$(foo)" unless you know why you need to leave them unquoted. If you leave them unquoted, the result is split into separate words wherever it contains whitespace, and each word is treated as a glob pattern. So unless the variable happens to contain a whitespace-separated list of glob patterns, put double quotes around it. Also, I recommend using $(…) rather than `…`; they are equivalent, except that nested quoting inside `…` is unreliable (also, ` is easily confused with ').

Do not parse the output of ls. If you need to act on all the files in a directory, the shell has a built-in construct that works: globbing. Instead of $(ls /path/to/directory), write /path/to/directory/*. This generates file names with the directory path; this is almost always what you need anyway, and if you don't, you can either call cd beforehand or strip away all or part of the directory. Below, I use ${f#*/*/}, which means $f with the shortest prefix matching */*/ stripped off.

for f in .templates/template_text/*; do
  g=$(cat "$f")
  h=$(cat ".templates/template_html/${f#*/*/}")
  find to_process/ -type f …

With find, you can use the simpler construct -exec, though -print0 combined with xargs -0 works too. Do not use xargs without -0, as it expects input quoted in a peculiar way which find does not produce.

find to_process/ -type f -exec perl … {} +

The next issue is that you're inserting the strings $f, $g and $h directly in your sed or perl regular expression. This is wrong: these variable do not contain a regular expression with the delimiter (/ in both cases) quoted. With sed, you'd need to do a pass of quoting on the strings, adding a backslash before any of /*.\[ in $f and before any of \&/ in $g and $h. With Perl, there is a simpler way: pass the values through the environment, and be sure to tell Perl that what you have is a string and not a regexp.

export f g h
find to_process/ -type f -exec perl -i -e '
    if ($n==0) {s/\Q$ENV{f}/$ENV{g}/; $n=1;}
    elsif ($n==1) {s/\Q$ENV{f}/$ENV{h}/; $n=2}}
' {} +
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.