Find duplicate paragraphs in two files and delete one

Question

I have two bib files, some of the entries are duplicates, the duplicates entries are in paragraphs, or could be identified with same pattern, e.g.

a.bib looks like

@InProceedings{Arranged,
  author = {Transcribed by hofei Arranged and by hofei},
  title  = {ALL OF ME},
  file   = {:All of Me.pdf:PDF},
  groups = {Solo Tab},
}

@InProceedings{P,
  author = {P and = and V V and V V},
  title  = {ANGELS WE HAVE HEARD ON HIGH Transcribed by hofei},
  file   = {:Angels We Have Heard on High.pdf:PDF},
  groups = {Solo Tab},
}

and b.bib

@InProceedings{Arranged,
  author = {Transcribed by hofei Arranged and by hofei},
  title  = {ALL OF ME},
  file   = {:All of Me.pdf:PDF},
  groups = {Solo Tab},
}

@InProceedings{,
  title  = {This Is My Father's World Standard Tuning Traditional Fast Tempo - “Thumbpicking” Style Arrangement by Mark Hanson},
  year   = {2005},
  file   = {:MyFathersWorld_p2.pdf:PDF},
  groups = {Solo Tab},
}

I know to show duplicates paragraph in two files is

$ awk -v RS=""  '{gsub(/\n/," "); print}' a.bib b.bib | sort | uniq -c | grep -vE '^\s*1 '
      2 @InProceedings{Arranged,   author = {Transcribed by hofei Arranged and by hofei},   title  = {ALL OF ME},   file   = {:All of Me.pdf:PDF},   groups = {Solo Tab}, }

But how to automatically delete the duplicate from b.bib?

Answer 1

It is the same as if we had lines, but now we have paragraphs. Parse two files, put in the hash the paragraphs of the first, and for the second print paragraph only if it doesn't exist.

awk -v RS="" -v ORS="\n\n" 'FNR==NR{a[$0]; next} !($0 in a)' a.bib b.bib

Output:

@InProceedings{,
  title  = {This Is My Father's World Standard Tuning Traditional Fast Tempo - “Thumbpicking” Style Arrangement by Mark Hanson},
  year   = {2005},
  file   = {:MyFathersWorld_p2.pdf:PDF},
  groups = {Solo Tab},
}

Be careful because a blank character anywhere could make it lose a duplicate. Perhaps you would like to run some diff together to confirm the results.