0

I have two bib files, some of the entries are duplicates, the duplicates entries are in paragraphs, or could be identified with same pattern, e.g.

a.bib looks like

@InProceedings{Arranged,
  author = {Transcribed by hofei Arranged and by hofei},
  title  = {ALL OF ME},
  file   = {:All of Me.pdf:PDF},
  groups = {Solo Tab},
}

@InProceedings{P,
  author = {P and = and V V and V V},
  title  = {ANGELS WE HAVE HEARD ON HIGH Transcribed by hofei},
  file   = {:Angels We Have Heard on High.pdf:PDF},
  groups = {Solo Tab},
}

and b.bib

@InProceedings{Arranged,
  author = {Transcribed by hofei Arranged and by hofei},
  title  = {ALL OF ME},
  file   = {:All of Me.pdf:PDF},
  groups = {Solo Tab},
}

@InProceedings{,
  title  = {This Is My Father's World Standard Tuning Traditional Fast Tempo - “Thumbpicking” Style Arrangement by Mark Hanson},
  year   = {2005},
  file   = {:MyFathersWorld_p2.pdf:PDF},
  groups = {Solo Tab},
}

I know to show duplicates paragraph in two files is

$ awk -v RS=""  '{gsub(/\n/," "); print}' a.bib b.bib | sort | uniq -c | grep -vE '^\s*1 '
      2 @InProceedings{Arranged,   author = {Transcribed by hofei Arranged and by hofei},   title  = {ALL OF ME},   file   = {:All of Me.pdf:PDF},   groups = {Solo Tab}, }

But how to automatically delete the duplicate from b.bib?

2
  • 2
    This seems more like a TeX question than a Unix question - see for example Merge two BibTeX files Sep 28 '20 at 15:56
  • 1
    That awk+sort+uniq+grep line is NOT how to show duplicate paragraphs between 2 files, it's trivial to do that with 1 call to awk alone. awk -v RS= 'NR==FNR{a[$0]; next} $0 in a' a.bib b.bib.
    – Ed Morton
    Sep 28 '20 at 19:24
1

It is the same as if we had lines, but now we have paragraphs. Parse two files, put in the hash the paragraphs of the first, and for the second print paragraph only if it doesn't exist.

awk -v RS="" -v ORS="\n\n" 'FNR==NR{a[$0]; next} !($0 in a)' a.bib b.bib

Output:

@InProceedings{,
  title  = {This Is My Father's World Standard Tuning Traditional Fast Tempo - “Thumbpicking” Style Arrangement by Mark Hanson},
  year   = {2005},
  file   = {:MyFathersWorld_p2.pdf:PDF},
  groups = {Solo Tab},
}

Be careful because a blank character anywhere could make it lose a duplicate. Perhaps you would like to run some diff together to confirm the results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.