I have a single source file which has multiple entries of this format:

some_id: some string

And a bunch of .txt files in different folders that have strings that are identified by id


How do I parse the source file in a way so that each string in .txt file which has id matching one of source file be added a new parameter like so

id="some_id" some_param="some string"

which is taken from value of corresponding source file ID


  • There can be multiple IDs in same file
  • There are never duplicated IDs in any file
  • They are never located on the same line
  • some_param is a literal
  • it does't matter where to add new some_param="string" as long as it is on the same line and spaced out

Some sample


apples: Apples are very tasty
grapes: Grapes kind of ok
trees: Gain mass from CO2
strawberry: Yummy


This is some plain text
Toreplace id="apples" string contains
The most vitamin-rich berry is id="strawberry"


And id="grapes" the


Or id="tress"

Given this sample, script should turn fruits/foo.txt into:

This is some plain text
Toreplace id="apples" string contains some_param="Apples are very tasty"
The most vitamin-rich berry is id="strawberry" some_param="Yummy"
  • elaborate your question: 1) could be there multiple ids in one file? 2) could they be located on the same line. Post some testable fragment – RomanPerekhrest Dec 7 '17 at 21:13
  • @RomanPerekhrest updated, not sure what you meant by testable fragment though. – knitevision Dec 7 '17 at 21:31
  • Where does the some_param come from, or is it a literal? – ilkkachu Dec 7 '17 at 21:37

Complex Grep + GNU Awk solution:

The crucial awk script content, let's say add_param_by_id.awk:

#!/bin/awk -f
BEGIN{ FS=":[[:space:]]+" }
NR==FNR{ a[$1] = $2; next }
match($0, /\<id="([^"]+)"/, b) && b[1] in a{ 
    sub(/\<id="[^"]+"/, "& some_param=\042"a[b[1]]"\042") 

The main commands:

export pat="($(cut -d':' -f1 source.yml | paste -s -d'|'))"
grep -ElZr "\<id=\"$pat\"" --include=*.txt | xargs -0 -I{} sh -c \
'awk -f add_param_by_id.awk source.yml "$1" > "/tmp/$$" && mv "/tmp/$$" "$1"' _ {} 

  • pat - variable containing regex alternation group with identifiers i.e. (apples|grapes|trees|strawberry)
  • grep -ElZr id=\"$pat\" --include=*.txt - prints all filenames which were matched for any of the specified identifiers

Example file content after processing:

This is some plain text
Toreplace string contains id="apples" some_param="Apples are very tasty"
The most vitamin-rich berry is id="strawberry" some_param="Yummy"
| improve this answer | |
  • RomanPerekhrest I'm a bit sorry, I have overlooked the complexity of the source files. Is there a way to modify the regex to actually include every line, white space and character that is after the match n up until (and excluding) id match n+1 ? Like here pastebin.com/KbuBJvbC – knitevision Dec 9 '17 at 1:00
  • @knitevision, NEW conditions is a pretext for a NEW question – RomanPerekhrest Dec 9 '17 at 7:09
  • that would be a duplicated question and closed for sure I suppose :( – knitevision Dec 9 '17 at 12:28

With certain caveats, such as "no strange characters in filenames", and "no / characters in matching or substitution data values" this should work for you. It's not terribly efficient, in that it makes one pass through every file it can find for each line in the YAML file, but to fix that I would want more information about the relative size of the YAML file to the number of target files.

while IFS=: read -r id sp
    id="id=\"$id\"" sp="some_param=\"${sp:1}\""

    find . -type f -name '*.txt' |
        while IFS= read -r file
            sed -i "/\<$id/s/$/ $sp/" "$file"
done <source.yml
| improve this answer | |
  • "no / characters in matching or substitution data values"... it's a lot more than that, you'll have to do some serious escaping if you want that sed command to work with arbitrary input (imagine what happens if some_param contains &), not to mention that two nested while..read loops to process text may result in the operator falling asleep... Also, I'm not sure why using * as a starting point for find... – don_crissti Dec 7 '17 at 23:27
  • @don_crissti, yes I know, thanks. The (other solution)[unix.stackexchange.com/a/409591/100397] here also fails on "unintended" RE matches. I was trying to highlight shortcomings of this solution. Perhaps I'd have been better to keep quiet as it will satisfy the question asked here. – roaima Dec 8 '17 at 5:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.