Extract part of file between first occurrences of two strings fast

Question

I have large (MB) text files containing a single line and need to extract the part between first occurrences of string1 and of string2. I found many recipes but most work for separate lines or extract to the last occurrence of the second string. The one I found that works using parameters infile, outfile, string1, string2 is:

#!/bin/bash

a="$(cat $1)"                   ## copy file1 to a

a="$(echo "${a#*"$3"}")"        ## cut up to (including) str1  

echo "${a%%"$4"*}" > $2         ## part after str1 up to str2

## where the last line may also be

printf '%s' "${a%%"$4"*}" > $2  ## part after str1 up to str2

but both versions are terribly slow (I have to repeat this many times per file). How speed up?

Answer 1

Some discussion in chat with the OP exposed that the single line was JSON from a network query. Pretty printing this with tools such as jq . < input_file or python -mjson.tool input_file made the file more suitable for conventional line by line processing.

Further discussion yielded that this was a bibliographic lookup, and the intention was to produce a flat file. The following jq program (my first) is deemed by the OP to be fast enough.

#!/bin/bash                                                                                                                           

jq -r  < "$1" '.response.docs[] |                                                                                          
    ("    Title: "+.title[]),                                                                                                          
    ("   Authors: "+(.author|join(""))),                                                                                               
    ("    Bibcode: "+.bibcode),                                                                                                        
    ("   AltBibcode: "+(.alternate_bibcode//[] | join(" : "))),                                                                        
    ("  "+(.abstract//"NOABSTRACT")),                                                                                                  
    ""' | fmt | sed 's/^    \?//'

There are a couple of subtle points. The desire is to have the abstract wrapped rather than one long line, so the output is sent through fmt. This program has the property that lines starting with different indentations are not joined together, hence the 4/3/4/3/2 pattern of spaces in the text being added such as " Title: ". fmt is then effectively just wrapping the abstract, indented by 2 spaces. The sed is then used to remove the 3 or 4 spaces that are on the other header lines. This is not perfect, very long lists of authors will get wrapped as well. A better sed program could join the authors back onto one line, remove blank AltBibcode entries etc.

The output format is designed so it can be post processed by an awk one line script

awk -vRS='' -vORS=$'\n\n' !/NOABSTRACT/'

to remove entries without abstracts.

Answer 2

$ sed -e 's/stringA/\n/;s/.*\n//;s/stringB.*//' file 

The removal of the code up to stringA is done in 2 stages (change stringA to newline and then remove the first line) rather than the more obvious "remove everything up to stringA" as the removal needs to be up to the first occurrence of stringA rather than the last. sed does not support non-greedy regular expressions, unlike many other tools such as python and perl.

$ perl -lpe '
    my($sa, $sb) = qw/stringA stringB/;
    my $p2 = index($_, $sb);
    my $p1 = index($_, $sa)+length($sa);
    $_ = substr($_, $p1, $p2-$p1);
'  file