source multiple files and output one file

Question

I have a file name "conf1" containing variables like:

name='john'
last=''
custom='1000'

and another file name conf2 like this:

name='john'
last='star'

I want to merge between them to one file but in a way that the merged file contain the variable in the same order I source them.

for example if I source conf1 and then conf2 the variable from conf 2 will override conf1. but I will also have the variable that I don't have in conf2.

I want to merge and create 1 file from both of them with only the variables that are unique and was sources last.

required output:conf3

name='john'
last='star'
custom='1000'

is this possible?

Answer 1

You could do:

$ awk -F= '{l[$1]=$0};END{for (i in l) print l[i]}' conf1 conf2
custom='1000'
last='star'
name='john'

Note that the order of the lines in the output is not guaranteed (based on how awk stores the array internally in a hash table), but settings in conf2 will override those in conf1.

where

• awk -F= ... conf1 conf2 call awk with = as separator on conf files.
• {l[$1]=$0} store definition of each var, newest overriding oldest
• END{ ... } in the end, (after all files processed)
• for (i in l) loop for all var,
• print l[i] and print it.

Answer 2

A Perl approach:

$ perl -F= -lane '$k{$F[0]}=$F[1]; END{print "$_=$k{$_}" for keys(%k)}' conf1 conf2 
last='star'
name='john'
custom='1000'

This is the same idea as Stéphane's answer. It builds a hash where the variable name is the key and its value is the value. Then, once all files have been processed, it prints out the key/value pairs. Note that the order of the lines is not guaranteed, and can change between invocations, but the values of conf2 will always override those of conf1.

Answer 3

For conf2 to take priority, it must precede conf1 in the sort args.

sort -suk1,1 -t= conf2 conf1 | tac

Output:

name='john'
last='star'
custom='1000'

The above does rely on the fact that the labelscustom, last and name sort appropriately for your required output order. However, when the labels are otherwise named, then this mod will work:

eval srx=\" 's/^'{name/1,last/2,custom/3}'=&/;' \"   # sort seq: 1,2,3,etc
sort -suk1,1 -t= conf2 conf1 | sed "$srx" | sort -n | cut -d= -f2-    

Answer 4

You could use join to merge the two files and preserve the order of the keys. Here's an example:
conf1:

name='john'
last=''
custom='1000'
rule='default'

conf2:

name='joe'
second='none'
fourth='dash'
last='star'
rule='whatever'

if you run:

{ { join -t= -1 2 -2 1 -o 1.1 1.2 2.2 <(sort -t= -k2,2 <(nl -ba -nrz -s= conf1)) \
<(sort -t= -k1,1 conf2); join -t= -1 2 -2 1 -v1 -o 1.1 1.2 1.3 \
<(sort -t= -k2,2 <(nl -ba -nrz -s= conf1)) <(sort -t= -k1,1 conf2); \
} | sort -k1,1n | cut -d= -f2-; join -t= -1 1  -2 2 -v2 -o 2.1 2.2 2.3 \
<(sort -t= -k1,1 conf1) <(sort -t= -k2,2 <(nl -ba -nrz -s= conf2)) \
| sort -k1,1n | cut -d= -f2-; } > merged_conf

the content of merged_conf is:

name='joe'
last='star'
custom='1000'
rule='whatever'
second='none'
fourth='dash'

Maybe it's easier to understand if working with two temporary files. First number the lines in conf1 and conf2 and sort them by 2nd field:

sort -t= -k2,2 <(nl -ba -nrz -s= conf1) >conf1.sorted
sort -t= -k2,2 <(nl -ba -nrz -s= conf2) >conf2.sorted

the above creates conf1.sorted:

000003=custom='1000'
000002=last=''
000001=name='john'
000004=rule='default'

and conf2.sorted:

000003=fourth='dash'
000004=last='star'
000001=name='joe'
000005=rule='whatever'
000002=second='none'

Now, the same command using the original files and the numbered/sorted files:

{ { join -t= -1 2 -2 1 -o 1.1 1.2 2.2 conf1.sorted <(sort -t= -k1,1 conf2); \
join -t= -1 2 -2 1 -v1 -o 1.1 1.2 1.3 conf1.sorted \
<(sort -t= -k1,1 conf2); } | sort -k1,1n | cut -d= -f2-; \
join -t= -1 1  -2 2 -v2 -o 2.1 2.2 2.3 <(sort -t= -k1,1 conf1) conf2.sorted \
| sort -k1,1n | cut -d= -f2-; } > merged_conf

Explanation:

join -t= -1 2 -2 1 -o 1.1 1.2 2.2 conf1.sorted <(sort -t= -k1,1 conf2)

joins the files based on field 2 from conf1.sorted and field 1 from conf2 printing the common lines but with the values (2.2) from conf2:

000002=last='star'
000001=name='joe'
000004=rule='whatever

and

join -t= -1 2 -2 1 -v1 -o 1.1 1.2 1.3 conf1.sorted <(sort -t= -k1,1 conf2)

prints the unpairable lines from conf1.sorted:

000003=custom='1000'

they're grouped {..} and the output is piped to sort -k1,1n | cut -d= -f2- so as to restore the order of keys from the original conf1:

name='joe'
last='star'
custom='1000'
rule='whatever'

then

join -t= -1 1  -2 2 -v2 -o 2.1 2.2 2.3 <(sort -t= -k1,1 conf1) conf2.sorted

joins based on field 1 from conf1 and field 2 from conf2.sorted printing unpairable lines from the latter:

000003=fourth='dash'
000002=second='none'

in the same manner, the output is piped to sort -k1,1n | cut -d= -f2- so as to restore the order of keys from the original conf2:

second='none'
fourth='dash'

Grouping all the above produces the desired output:

name='joe'
last='star'
custom='1000'
rule='whatever'
second='none'
fourth='dash'

this is then redirected (>) to merged_conf. Finally, remove the sorted files:

rm conf{1,2}.sorted

This could then be repeated for additional config files using merged_conf and e.g conf3 as arguments and so on...

Answer 5

Use this:

name=${name:-John}
last=${last:-}
custom=${custom:-1000}

... and similarly in the other file. Read your shell's manual page. This will set the variables to the specified values unless they are already set and non-null. It will not matter what order you source the files, at least not with the example data that you showed.