I am using bash and trying to employ sed command for matching a string and replace it with another between two files.

Objective: To replace all strings[sp_*] by matching them from another file containing [sp_* Var_Names]. Please be noted that: 1. The order is synchronized but not consecutive in both files, so cannot use paste. 2. Functionality is similar to primeKey operations in mysql.

File 1

|     +--sp_O18983_
|  +--| (52)
|  |  |  +--sp_Q9BDS6_
|  |  +--| (26)
|  |     |  +--sp_O19024_
|  |     +--| (29)
|  |        +--sp_Q9XT45_



Purpose: To sed -ie 's/O00574/CXCR6_HUMAN/g' File1

inline Bash script:

cat File2 | while read id; do upID=`echo $id | cut -d " " -f1`; upName=`echo $id | cut -d " " -f2`; sed -ie 's/sp_$upID/$upName/g' File1; done



cat File2 | while read id;
    upID=`echo $id | cut -d " " -f1`
    upName=`echo $id | cut -d " " -f2`

    sed -ie 's/sp_$upID/$upName/g' File1

Problem: The sed command does not work in the loop. No change is observed in File1, at all. If I echo the sed command from the script and then run it in the terminal it works as expected. I cannot figure out what could be the problem.

Thank you for your valuable comments and solution.

  • 3
    You are using ' ' quotes in your sed statement which prevents variable expansion. Use " " quotes instead. Also you can make it simpler with "while read upID upName" and not need the two cut statements. Jun 2 '16 at 15:38
  • Dear Stephen, Thanks for the error and added information. It would be very useful in solving many other cases. Thanks :) Somehow, I was saturated to see the single quotes. ;P Jun 2 '16 at 16:06
  • For the "while upId upName" the upName overwrites the upId everytime. Jun 2 '16 at 16:08

generate a sed script from your index file (File2) instead of a loop then run that script against your File1.. It will be MUCH faster :).

 awk '{ print "s/sp_"$1"/"$2"/g"}' File2.txt > tranform.sed

then do:

 sed -i -f transform.sed File1.txt 

so your entire script could be:

awk '{ print "s/sp_"$1"/"$2"/g"}' File2.txt > transform.sed
sed -f transform.sed File1.txt

## if you want to remove your transformation file
rm transform.sed
  • 1
    Nice, but if this is scripted you shouldn't use a fixed temp file name. Since the Original Poster specified bash, I'd opt for process substitution instead: sed -i -f <(awk '{print "s/sp_"$1"/"$2"/g"}' File2.txt) File1.txt
    – Wildcard
    Jun 2 '16 at 23:36
  • So, you use the powerful awk to generate an input file for the less powerful sed, instead of using awk directly? Jun 3 '16 at 7:42
  • The real question is does it work? :) There is always ten different ways to do something. awk is neither better or worse than sed. They are tools to get something done. Can you do it is awk in two easy to read lines?
    – Rob
    Jun 3 '16 at 10:45
  • Thanks everyone for the response. As pointed out in the comments by +Stephen Harris, the problem was with the single quoting 's///g'. With double quotes it works perfectly fine. If I try the awk + sed it takes longer than the doing while directly on the file and sed -i. Jun 3 '16 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.