Script to extract selected entries from a bibtex file

Question

I have a large bibtex file with many entries where each entry has the general structure

@ARTICLE{AuthorYear,
item = {...},
item = {...},
item = {...},
etc
}

(in some cases ARTICLE might be a different word e.g. BOOK)

What I would like to do is write a simple script (preferably just a shell script) to extract entries with given AuthorYear and put those in a new .bib file.

I can imagine that I can recognize the first sentence of an entry by AuthorYear and the last by the single closing } and perhaps use sed to extract the entry, but I don't really know how to do this exactly. Can someone tell me how I would achieve this?

It should probably be something like

sed -n "/AuthorYear/,/\}/p" file.bib

But that stops due to the closing } in the first item of the entry thus giving this output:

@ARTICLE{AuthorYear,
item = {...},

So I need to recognize whether the } is the only character at a line and only have 'sed' stop reading when this is the case.

Answer 1

I would recommend using a language with a battle-tested BibTeX library instead of reinventing that wheel. For example

#!/usr/bin/env perl
use strict;
use warnings;
use autodie;
use BibTeX::Parser;

open my $fh, '<', $ARGV[0];
my $parser = BibTeX::Parser->new($fh);
my @authoryear;
while (my $entry = $parser->next) {
    if ($entry->parse_ok) {
        if ($entry->key eq "AuthorYear") {
            push @authoryear, $entry;
        }
    }
    else {
        warn "Error parsing file: " . $entry->error;
    }
}

# I'm not familiar with bibtex files, so this may be insufficient
open my $out, '>', "authoryear.bib";
foreach my $entry (@authoryear) {
    say $out $entry->raw_bibtex;
}

You'll probably have to install the module: cpan install BibTeX::Parser

Answer 2

The following Python script does the desired filtering.

#!/usr/bin/python
import re

# Bibliography entries to retrieve
# Multiple pattern compilation from: http://stackoverflow.com/a/11693340/147021
pattern_strings = ['Author2010', 'Author2012',]
pattern_string = '|'.join(pattern_strings)
patterns = re.compile(pattern_string)


with open('bibliography.bib', 'r') as bib_file:
    keep_printing = False
    for line in bib_file:
        if patterns.findall(line):
            # Beginning of an entry
            keep_printing = True

        if line.strip() == '}':
            if keep_printing:
                print line
                # End of an entry -- should be the one which began earlier
                keep_printing = False

        if keep_printing:
            # The intermediate lines
            print line,

Personally, I prefer moving to a scripting language when the filtering logic becomes complex. That, perhaps, has an advantage on the readability factor at least.

Answer 3

Now we also have the Python bibparsing module, that allows to analyze BibTeX databases with Python. For example I use the following script to calculate the number of authors in collaborative papers:

#!/usr/bin/python
import sys
import bibtexparser as bp
with open(sys.argv[1]) as bibtex_file:
    bd = bp.load(bibtex_file)
    for art in bd.entries_dict:
    print("*********")
    ae = bd.entries_dict[art]
    print(ae[u'title'])
    auths=ae[u'author'].split(" and ")
    print(len(auths))
    print(auths[0]+" --- "+auths[-1])

Answer 4

Another option would be to use bibtool.

Example:

bibtool -- select{$key ”AuthorYear”} input.bib -o output.bib

Check out the manual for specific cases.

Answer 5

I made an expanded version of Michels own answer that takes bib-file with lots of entries and splits it up into one bib-file per entry. this could easily be expanded to other purposes.

Most of this is is a straightforward implementation of Michels answer into a do-loop. However, I added a comma within the bash-expansion:

 sed -n "/"${entry},"/, /^ *\}/p" work-bibliotek.bib 

instead of

sed -n "/"${entry}"/, /^ *\}/p" work-bibliotek.bib 

otherwise an entry like

 author2010fo 

will include the entry for

author2010fobar 

This caused me quite a bit of headaches today, finding that error. So perhaps I just saved someone else those hours instead :)

I also included a test to see if all entries are unique, and it excudes @comments made by jabref.

#!/bin/bash
# bibliotek-entries-for-sig.sh


cd /home/emil/Dropbox/bibliotek


# get all entries
entries=$(rg '@' work-bibliotek.bib | rg -v '@Comment' | grep -Po '(?<={)\w+')

nonuniques=$(echo "$entries" | wc -l)
uniques=$(echo "$entries" | uniq  | wc -l)




if [ $nonuniques != $uniques ]; then 
    # echo "hey"
    notify-send "citekeys not unique. exitting" --urgency critical 
    exit

else

    # remove old ones
    rm singles-entries/work-bibliotek-entries/*.bib

echo "$entries"



    for entry in $entries; do
     
     # the comma is important otherwise 
     # author2010fo 
     # will be the same as
     # author2010fobar 
     sed -n "/"${entry},"/, /^ *\}/p" work-bibliotek.bib > "singles-entries/work-bibliotek-entries/$entry.bib"
    done


fi 

Answer 6

This is a Bash script which reads each line and uses regex matching to extract each entry that has the required pattern in its head. You can call it getbibs or something:

#!/usr/bin/env bash
# usage: ./getbibs pattern input.bib output.bib

while read entry; do
    if [[ $entry =~ ^@.*{$1,$ ]]; then
        printf "%s\n" "$entry" >> "$3"
        while read item; do
            [[ $item =~ ^@.*$ ]] && break
            printf "%s\n" "$item" >> "$3"
        done
    fi
done < "$2"

To extract all entries with an author year of 1989 you could do:

$ chmod +x ./getbibs
$ ./getbibs 1989 file.bib author.bib

It might have some issues that I haven't tested yet, but it seems to work okay for the task.

Answer 7

Just to be complete, the way I figured out myself, not as nice as some of the others, but it works:

entries=( AuthorYear1 AuthorYear2 )
for entry in "${entries[@]}" do
     sed -n "/"${entry}"/, /^ *\}/p" refs.bib 
done

It can be run from the commandline or put in a bash script.