1

I have 3 text files. I want to search file3 for a string in file2 and replace it with a string in file1 in found. I need to append a custom tag from file1 to the end of the string in file3, replacing the partial string found from file2.

file3

aws ec2 create-tags --region us-east-1 --resourcesi-XXXXX --tags Key=Developer Name,Value=XXXXX Key=Resource Group,Value=arn:aws:iam::XXXXX:root
aws ec2 create-tags --region us-east-1 --resourcesi-XXXX --tags Key=Developer Name,Value=XXXXX Key=Resource Group,Value=arn:aws:iam::XXXXX:user/user

file2

arn:aws:iam::XXXXX:root 
arn:aws:iam::XXXXX:user/user

file1

my_custom_tag_1
my_custom_tag_2

Desired output:

aws ec2 create-tags --region us-east-1 --resourcesi-XXXXX --tags Key=Developer Name,Value=XXXXX Key=Resource Group,Value=my_custom_tag_1
aws ec2 create-tags --region us-east-1 --resourcesi-XXXX --tags Key=Developer Name,Value=XXXXX Key=Resource Group,Value=my_custom_tag_2

I've tried loading the lines from the file into an array and including the index in a sed replace.

sed "s|${file2array[0]}|${file1array[0]}|g" file3.txt

But this returns a "no previous regular expression" error. I've also tried writing the array indexes to unique variables with a for loop and using the same approach above with the variables

sed "s|$var2|$var1|g" file3.txt

This also fails

Interestingly,

sed "s|${file2array[0]}|customtext}|g" file3.txt

fails but

sed "s|customtext|${file1array[0]}|g" file3.txt

succeeds.

Any help is greatly appreciated. Been working on this for dozens of hours now.

marked as duplicate by don_crissti, mdpc, Eric Renouf, G-Man, Anthon Sep 22 '16 at 2:34

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • "no previous regular expression" implies your shell variables are empty. – glenn jackman Sep 21 '16 at 19:54
1

Try:

awk 'FNR==NR{a[FNR]=$0; next} NR<=length(a)+FNR{b[FNR]=$0; next} {for (i=1;i<=length(a);i++) gsub(a[i], b[i])} 1' file2 file1 file3

For example:

$ awk 'FNR==NR{a[FNR]=$0; next} NR<=length(a)+FNR{b[FNR]=$0; next} {for (i=1;i<=length(a);i++) gsub(a[i], b[i])} 1' file2 file1 file3
aws ec2 create-tags --region us-east-1 --resourcesi-XXXXX --tags Key=Developer Name,Value=XXXXX Key=Resource Group,Value=my_custom_tag_1
aws ec2 create-tags --region us-east-1 --resourcesi-XXXX --tags Key=Developer Name,Value=XXXXX Key=Resource Group,Value=my_custom_tag_2

How it works

  • FNR==NR{a[FNR]=$0; next}

    This saves all the lines in file2 in array a.

    FNR is the number of lines read from the current file. NR is the number of lines read in total. Thus, if FNR==NR, we are reading the first named file, file2. a[FNR]=$0 adds the current line, denoted $0, into array a under the key FNR.

    The command next tells awk to skip the remaining commands and start over on the next line.

  • NR<=length(a)+FNR{b[FNR]=$0; next}

    This saves all the lines of file1 in array b.

    Here, we use a similar test, NR<=length(a)+FNR, to determine if we are reading the second file. b[FNR]=$0 adds the current line, denoted $0, into array b under the key FNR.

    The command next tells awk to skip the remaining commands and start over on the next line.

  • for (i=1;i<=length(a);i++) gsub(a[i], b[i])

    If we get here, we are reading the third file. This replaces any text matching a line in file2 with the corresponding text from file1.

    The loop for (i=1;i<=length(a);i++) loops over the line number of every line in array a.

    gsub(a[i], b[i]) replaces any occurrence of text a[i] with the text b[i].

    Note that the text in file2 is treated as a regular expression. If you need to have any regex-active characters in this file, they should be escaped.

  • 1

    This is awk's cryptic short-hand for print-the-line.

  • Thank for the well thought out answer. Replacing the generic file names at the end with the corresponding file names it doesn't seem to work. It just echos the file3 file to the screen. Let me know if I have overlooked something in your answer. Truly appreciate the help. – Jeff Carson Sep 21 '16 at 4:41
  • @JeffCarson First, did it work for you on the test files shown in your question? As you can see above, it worked for me. "Replacing the generic file names at the end with the corresponding file names it doesn't seem to work." It is important that the files be in the same order that are specified here: file2 file1 file3. The first file specified, file2, is the one with the text that we want to remove. – John1024 Sep 21 '16 at 4:46
  • Actually, file3 (the very last part of each line) is what I am needing to remove and replace with a line from file1. File 2 holds the string that we are searching for in file3. If that string is found (which will be a part of the line in file 3) I want to delete that match and replace with a line from file1, or the custom tag. Does that make sense? – Jeff Carson Sep 21 '16 at 4:49
  • @JeffCarson Did it work for you, as it did for me, on the test files shown in your question? – John1024 Sep 21 '16 at 4:50
  • 1
    Apologies. I've worked on it for hours before I ask, and one minute after I ask I find the issue. The second occurrence was missing a letter, hence not an exact match. Thank you once again. – Jeff Carson Sep 29 '16 at 13:34
0

I'd write

awk '
  BEGIN { FS = OFS = "=" }
  FILENAME == "file1" {
    tag[FNR] = $0
  }
  FILENAME == "file2" {
    str[$0] = FNR
  }
  FILENAME == "file3" {
    if ($NF in str) $NF = tag[str[$NF]]
    print
  }
' file1 file2 file3

I think it's pretty straightforward. Let me know if you have questions.

Not the answer you're looking for? Browse other questions tagged or ask your own question.