Compare lines and upgrade two different files

Question

I have a couple of text files with the following features:

$ cat file_1
Line A
Line B
Line C
Line D

$ cat file_2
Line A
Line added 1
Line B
Line D
Line added 2

They are such that file_1 has some lines that file_2 does not contain and vice-versa. I would like to upgrade each other with the missing lines so that both will become

Line A
Line added 1
Line B
Line C
Line D
Line added 2

The order of file_1 is preserved, but with the integrations coming from file_2 put in the same places as in file_2 (not in the head or in the tail or in random positions).

1) Is it possible to merge this way the files through an appropriate bash script?

2) Is it possible to do the same, when instead of lines I have paragraphs, that is: blocks of lines?

Answer 1

diff file_1 file_2 | grep -Ev '^<|[0-9]+d[0-9]+' | patch file_1

Answer 2

1) Is this possible to merge this way the files through an appropriate bash script?

As other commenters have said, telling apart add/move/delete is not precise, and best left to diff. But with the restriction that the "original" lines are unique, and appear in the same order in both files, it's more straightforward:

#!/bin/bash

#build list of common lines
grep -Fxf file_2 file_1 > common

#optional: confirm that they appear in the same order in both files
grep -Fxf file_1 file_2 > common2
if ! diff -q common common2 ; then
    echo "Duplicate or rearranged common lines, can't merge" >&2
    exit 1
fi

#copy lines from input until one is found that matches the argument
function copy_till () {
    while read l && [ "$l" != "$1" ] ; do
            printf "%s\n" "$l"
    done
}

# open both files, for parallel reading
exec 3< file_1
exec 4< file_2

#for each line in the common file
while read line ; do
    # copy any lines that were inserted before it, for each input file
    copy_till "$line" <&3
    copy_till "$line" <&4
    # and the original line
    printf "%s\n" "$line"
done < common > merged

# any trailing lines, after the last common line
cat <&3 >> merged
cat <&4 >> merged

2) Is it possible to do the same, when instead of lines I have paragraphs, that is: blocks of lines?

Once you have a way to do this for lines, you can use sed to turn paragraphs to long lines (saving the newlines as some other temporary token) and back. Borrowing heavily from the annotated script in https://unix.stackexchange.com/a/219562/90751 :

sed '/^$/!{H;$ba;d};:a;x;s/\n/\\n/gp;d' -i.bak file_1
sed '/^$/!{H;$ba;d};:a;x;s/\n/\\n/gp;d' -i.bak file_2
merge.sh # or whatever you called the answer to part 1)
sed 's/\\n/\n/g' merged > merged.paras

mv file_1.bak file_1
mv file_2.bak file_2

If the string \n appears in your paragraphs, use another string as the newline token.

Answer 3

1) Is it possible to merge this way the files through an appropriate bash script?

Provided:

• file_2 is basically file_1 with lines added and deleted but not moved around, and
• neither file contains tab characters,

GNU diff provides the key to a simple solution:

#!/usr/bin/env bash
diff -y "$1" "$2" | while IFS=$'\t' read s1 s2 s3; do
  if [[ $s1 == *\> ]]; then
    # New line from second file
    echo "$s2"
  else
    case "$s2" in
      *\|) # Changed line, so...
        echo "$s1"
        echo "$s3"
      ;;
      *) # Output first file
        echo "$s1"
      ;;
    esac
  fi
done

That send the unified output to stdout, where you can do whatever you want with it. I suggest inspecting the output first, instead of blindly overwriting both file_1 and file_2.

(If you're wondering how this works, run diff -y file_1 file_2 | cat -A to see what's really in the diff output.)

2) Is it possible to do the same, when instead of lines I have paragraphs, that is: blocks of lines?

Yes. The basic logic is the same as above, but you have to first convert paragraphs to single lines and use the reformatted texts as the inputs to diff above. You then have two options for output:

• Preserve original paragraphs. Read each paragraph from the correct file and output that. That's an exercise for you.
• Generate new paragraphs. Each echo in the script above becomes echo ... | fmt -w<desire_line_width>.

Answer 4

This is a problem of essentially performing a 2-way or bi-directional merge between two files syncing the differences between them. My own use-case for doing this was performing upgrades on a GitHub repo and retaining unique data in a variables file to preclude re-keying it on every upgrade!

I'll show you the expression first and the test data used so you can recreate the results to validate (just change the field separator from '=' to an empty space ' ' for your data set):

 paste -d'\n' file1.txt file2.txt|awk -F'=' '!seen[$1]++' > file3.txt

My test data looks like this:

file1.txt:

LineA='value1'
LineB='value2'
LineC='value3'
LineD='value4'
#
LineE='value5'
LineF='value6'
#
LineG='value7'
#
LineH='value8'

file2.txt:

LineA=''
LineB=''
NEWVARIABLE1='This only Exists in file2.txt Under LineB'
LineC=''
LineD=''
#
LineE=''
NEWVARIABLE2='This only Exists in file2.txt Under LineE'
LineF=''
#
LineG=''
#
LineH=''
NEWVARIABLE3='This only Exists in file2.txt under LineH'

Output of:

paste -d'\n' file1.txt file2.txt|awk -F'=' '!seen[$1]++' > file3.txt

Combined file3.txt looks like this:

LineA='value1'
LineB='value2'
LineC='value3'
NEWVARIABLE1='This only Exists in file2.txt Under LineB'
LineD='value4'
#
LineE='value5'
LineF='value6'
NEWVARIABLE2='This only Exists in file2.txt Under LineE'
LineG='value7'
LineH='value8'

NEWVARIABLE3='This only Exists in file2.txt under LineH'

Remark that in file2.txt all the values are empty ('') except for NEWVARIABLEx values. Reviewing the output you can see all the unique data from file1.txt has been preserved after the merge.

Also remark that each new "variable" downshifts added to file2.txt (the "updated" file) by 1 line in the combined file3.txt. So if you add 4 new variables to file2.txt, in the combined file3.txt the FOURTH new variable will be 4 lines lower in file3.txt. In my use case this is not a problem however.

Sadly, this will work on your FIRST use case- doing the 2-way file merge. However, if you added a continuous block of test to file2.txt ( again, the "updated file) these would be interleaved with the surrounding lines from file1.txt in the combined file3.txt. So I got you half the way there but my use case only overlaps yours on the first part of your question.

I tried tons of different solutions offered by other posters on this forum and others, but this was really the only one that did the business. HTH-