1

I have large (MB) text files containing a single line and need to extract the part between first occurrences of string1 and of string2. I found many recipes but most work for separate lines or extract to the last occurrence of the second string. The one I found that works using parameters infile, outfile, string1, string2 is:

#!/bin/bash

a="$(cat $1)"                   ## copy file1 to a

a="$(echo "${a#*"$3"}")"        ## cut up to (including) str1  

echo "${a%%"$4"*}" > $2         ## part after str1 up to str2

## where the last line may also be

printf '%s' "${a%%"$4"*}" > $2  ## part after str1 up to str2

but both versions are terribly slow (I have to repeat this many times per file). How speed up?

Improve this question
8
  • Is there some character in the single line that occurs a lot but isn't in your patterns that you can use tr to convert to a newline? If so then you can use one of the solutions you have found. I am thinking about perhaps a tab separated or comma separated set of fields? You should change a="$(echo "${a#*"$3"}")" to a=${a#*"$3"} to avoid creating subshells and copies of the data. – icarus Oct 6 '19 at 14:24
  • @icarus: change to a=${a#*"$3"} didn't speed up. Breaking up into lines: I have been trying that but without success, still cannot just skip lines but need to skip everything between strings. No single divider character. I could split into many files with },{ as divider pattern and treat them all this way, is that faster? – Rob Rutten Oct 6 '19 at 14:44
  • What characters are in string 1 and string 2? Is this JSON? – muru Oct 6 '19 at 14:47
  • I don't have the stetup to test what would actually speed things up, and I don't think I'll have time to figure out all the details today, but I suspect you could speed things up by compressing your operation into a single stream of grep or other regex statements? This suggests grep -oE and this shows how to do capture groups. Of course a compiled regex would be ideal. – ShapeOfMatter Oct 6 '19 at 14:48
  • I had to google JSON. I put an example (a relatively small case) at staff.science.uu.nl/~rutte101/rrlnk/ads-suemoto.txt. I extract per field between {...},{....} the title, author, bibcode fields. A long sequence of mostly sed commands. I will look into this stream suggestion - new to me. Compiled regex??? Guess I need to learn. – Rob Rutten Oct 6 '19 at 15:00
1

Some discussion in chat with the OP exposed that the single line was JSON from a network query. Pretty printing this with tools such as jq . < input_file or python -mjson.tool input_file made the file more suitable for conventional line by line processing.

Further discussion yielded that this was a bibliographic lookup, and the intention was to produce a flat file. The following jq program (my first) is deemed by the OP to be fast enough.

#!/bin/bash                                                                                                                           

jq -r  < "$1" '.response.docs[] |                                                                                          
    ("    Title: "+.title[]),                                                                                                          
    ("   Authors: "+(.author|join(""))),                                                                                               
    ("    Bibcode: "+.bibcode),                                                                                                        
    ("   AltBibcode: "+(.alternate_bibcode//[] | join(" : "))),                                                                        
    ("  "+(.abstract//"NOABSTRACT")),                                                                                                  
    ""' | fmt | sed 's/^    \?//'

There are a couple of subtle points. The desire is to have the abstract wrapped rather than one long line, so the output is sent through fmt. This program has the property that lines starting with different indentations are not joined together, hence the 4/3/4/3/2 pattern of spaces in the text being added such as " Title: ". fmt is then effectively just wrapping the abstract, indented by 2 spaces. The sed is then used to remove the 3 or 4 spaces that are on the other header lines. This is not perfect, very long lists of authors will get wrapped as well. A better sed program could join the authors back onto one line, remove blank AltBibcode entries etc.

The output format is designed so it can be post processed by an awk one line script

awk -vRS='' -vORS=$'\n\n' !/NOABSTRACT/'

to remove entries without abstracts.

Improve this answer
1
  • If there are a lot of authors or long titles then changing 's/^ \?//' to '/:/s/^ \?//' can keep the continuation lines indented, at least until they start containing colons. – icarus Oct 8 '19 at 16:57
0
$ sed -e 's/stringA/\n/;s/.*\n//;s/stringB.*//' file

The removal of the code up to stringA is done in 2 stages (change stringA to newline and then remove the first line) rather than the more obvious "remove everything up to stringA" as the removal needs to be up to the first occurrence of stringA rather than the last. sed does not support non-greedy regular expressions, unlike many other tools such as python and perl.

$ perl -lpe '
    my($sa, $sb) = qw/stringA stringB/;
    my $p2 = index($_, $sb);
    my $p1 = index($_, $sa)+length($sa);
    $_ = substr($_, $p1, $p2-$p1);
'  file
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.