Some discussion in chat with the OP exposed that the single line was JSON from a network query. Pretty printing this with tools such as jq . < input_file
or python -mjson.tool input_file
made the file more suitable for conventional line by line processing.
Further discussion yielded that this was a bibliographic lookup, and the intention was to produce a flat file. The following jq
program (my first) is deemed by the OP to be fast enough.
#!/bin/bash
jq -r < "$1" '.response.docs[] |
(" Title: "+.title[]),
(" Authors: "+(.author|join(""))),
(" Bibcode: "+.bibcode),
(" AltBibcode: "+(.alternate_bibcode//[] | join(" : "))),
(" "+(.abstract//"NOABSTRACT")),
""' | fmt | sed 's/^ \?//'
There are a couple of subtle points. The desire is to have the abstract wrapped rather than one long line, so the output is sent through fmt
. This program has the property that lines starting with different indentations are not joined together, hence the 4/3/4/3/2 pattern of spaces in the text being added such as " Title: ". fmt is then effectively just wrapping the abstract, indented by 2 spaces. The sed is then used to remove the 3 or 4 spaces that are on the other header lines. This is not perfect, very long lists of authors will get wrapped as well. A better sed program could join the authors back onto one line, remove blank AltBibcode entries etc.
The output format is designed so it can be post processed by an awk one line script
awk -vRS='' -vORS=$'\n\n' !/NOABSTRACT/'
to remove entries without abstracts.
tr
to convert to a newline? If so then you can use one of the solutions you have found. I am thinking about perhaps a tab separated or comma separated set of fields? You should changea="$(echo "${a#*"$3"}")"
toa=${a#*"$3"}
to avoid creating subshells and copies of the data.grep -oE
and this shows how to do capture groups. Of course a compiled regex would be ideal.