0

Let's say I have the following file:

A random Title 1
BLOCK
1- a block of text that can contain any character
and it  also can contain multiple lines
BLOCK

A random Title 2
BLOCK
2- a block of text that can contain any character
and it  also can contain multiple lines
BLOCK

A random Title 3
BLOCK
3- a block of text that can contain any character
and it  also can contain multiple lines
BLOCK

This file can have multiple blocks of text like these ones. I'd like to distribute the parameters from this text on the following JSON:

[
    {
        "title": "A random Title 1",
        "body": "1- a block of text that can contain any character\nand it  also can contain multiple lines"
    },
    {
        "title": "A random Title 2",
        "body": "2- a block of text that can contain any character\nand it  also can contain multiple lines"
    },
    {
        "title": "A random Title 3",
        "body": "3- a block of text that can contain any character\nand it  also can contain multiple lines"
    }
]

I know I could solve this problem by creating a loop that goes char by char inside this file and then I could create the logic to divide all the variables properly inside the JSON. But I'm wondering if there's a more simple solution using the command line. Can I use AWK for distributing parameters that I get inside a file on a JSON output? Or am I missinterpreting the functionality of AWK in this case?

4
  • AWK doesn't natively understand JSON, so in the general case it would probably be painful to use AWK to parse and manipulate JSON documents. If the use case was simple enough it might not be so bad. But I'd recommend looking into something like jq instead. Altough even jq might be tedious to use if you are merging data into a JSON document. In that case using Python, Node, Ruby, etc... will probably be easier in the end.
    – cnamejj
    Jun 18, 2021 at 23:28
  • @cnamejj jq is a tool for reading JSON files... But in this case I'm doing the other way around, I need a tool that reads and interprets plain text while converting it to JSON...
    – raylight
    Jun 18, 2021 at 23:31
  • 1
    You can use jq to generate JSON too, but I agree it's not ideal for what you're talking about. So I'll expand my "If the use case was simple enough it might not be so bad." comment from above... If you have a very clean plain text input files (meaning no odd characters that might break the JSON, consistent formatting, etc...) then using AWK like an engine to use a template to transform the lines into JSON shouldn't be too bad. But if the input isn't super clean, it will be a pain to add the parsing/checking in AWK.
    – cnamejj
    Jun 18, 2021 at 23:32
  • You should have included a double quote plus any other potentially problematic characters (if there are any) in some of the title and body sections of your examples to make sure that any potential solution handles them correctly.
    – Ed Morton
    Jun 19, 2021 at 1:08

5 Answers 5

3

You could try Miller

$ mlr --inidx --irs '\n\n' --ifs 'BLOCK' --ojson --jvstack --jlistwrap \
    put -S 'for(k,v in $*){$[k] = strip(v)}' then \
    cut -f 1,2 then \
    rename '1,title,2,body' file
[
{
  "title": "A random Title 1",
  "body": "1- a block of text that can contain any character\nand it  also can contain multiple lines"
}
,{
  "title": "A random Title 2",
  "body": "2- a block of text that can contain any character\nand it  also can contain multiple lines"
}
,{
  "title": "A random Title 3",
  "body": "3- a block of text that can contain any character\nand it  also can contain multiple lines"
}
]

You can prettify the output by piping it through jq '.' - or omit the --jvstack --jlistwrap options and pipe through jq -s '.':

$ mlr --inidx --irs '\n\n' --ifs 'BLOCK' --ojson \
    put -S 'for(k,v in $*){$[k] = strip(v)}' then \
    cut -f 1,2 then 
    rename '1,title,2,body' file | jq -s '.'
[
  {
    "title": "A random Title 1",
    "body": "1- a block of text that can contain any character\nand it  also can contain multiple lines"
  },
  {
    "title": "A random Title 2",
    "body": "2- a block of text that can contain any character\nand it  also can contain multiple lines"
  },
  {
    "title": "A random Title 3",
    "body": "3- a block of text that can contain any character\nand it  also can contain multiple lines"
  }
]

The cut -f 1,2 is only necessary because the second BLOCK marker implies a third (empty) field - it could be replaced by verb remove-empty-columns if you prefer (although the latter is non-streaming).


Dealing with contiguous newlines in the body

Unfortunately the above does not distinguish between contiguous newlines as the input record separator and contiguous newlines that may occur between BLOCK ... BLOCK body delimiters. As a workaround, you could pre-process the input to replace newlines in the body by \n sequences, then substitute back to literal newlines before writing out the JSON (where they will be escaped back to \n by Miller):

sed '/^BLOCK/{:a;N;/BLOCK$/!ba;s/\n/\\n/g;}' file | 
  mlr --inidx --irs '\n\n' --ifs 'BLOCK' --ojson put -S '$2 = gsub($2,"\\n","\n"); for(k,v in $*){$[k] = strip(v)}' then cut -f 1,2 then rename '1,title,2,body'

You could pass the sed filter to Miller as a --prepipe command, but the quoting becomes tricky.

4
  • I've found an edge case when I use empty lines inside 'BLOCK'... In this case, this command will divide the information inside a single 'BLOCK' into different arrays ... Do you know if a workaround for this issue using miller would be simple?
    – raylight
    Jun 19, 2021 at 18:39
  • 1
    @raylight sorry no I don't - AFAIK there's no way for miller's nidx format to distinguish between multiple contiguous empty lines inside a record versus multiple contiguous empty lines as a record separator. You'd need some kind of context-aware parser for that. Jun 19, 2021 at 18:46
  • 1
    @raylight however I have added a workaround that involves a pre-processing step Jun 19, 2021 at 21:54
  • @raylight if the text can contain newlines, and you already told us the text can contain any char (and so by that definition could itself contain a line that's BLOCK which is one reason I didn't use it as a delimiter in my answer) how can you robustly tell where the text blocks start/end?
    – Ed Morton
    Jun 20, 2021 at 14:22
2

In the TXR language, we could do it like this:

$ txr data.txr data
[{"title":"A random Title 1","body":"1- a block of text that can contain any character\nand it  also can contain multiple lines"},
 {"title":"A random Title 2","body":"2- a block of text that can contain any character\nand it  also can contain multiple lines"},
 {"title":"A random Title 3","body":"3- a block of text that can contain any character\nand it  also can contain multiple lines"}]

Where the code in data.txr is:

@(bind vec @(vec))
@(repeat)
@title
BLOCK
@(collect)
@lines
@(until)
BLOCK
@(end)
@(cat lines "\n")
@(do (vec-push vec #J^{"title" : ~title, "body" : ~lines}))
@(end)
@(do (put-jsonl vec))

We build up a vector of hashes: the underlying data structure corresponding to the desired JSON.

The #J prefix indicates a JSON literal embedded in Lisp. Here, we have a ^ which indicates the literal is being quasiquoted; ~ characters indicate the unquotes which insert values into the template: the title, and the expression that calculates the body from the collected lines catenating strings with a newline.

put-jsonl means put-json, with a newline after it. By default, on the *stdout* stream.

Indentation is recommended, which looks like this:

@(bind vec @(vec))
@(repeat)
@  title
BLOCK
@  (collect)
@    lines
@  (until)
BLOCK
@  (end)
@  (cat lines "\n")
@  (do (vec-push vec #J^{"title" : ~title, "body" : ~lines}))
@(end)
@(do (put-jsonl vec))

It could be done with a sort of Awk; the Awk macro in TXR Lisp:

$ txr data.tl data
[{"title":"A random Title 1","body":"1- a block of text that can contain any character\nand it  also can contain multiple lines"},
 {"title":"A random Title 2","body":"2- a block of text that can contain any character\nand it  also can contain multiple lines"},
 {"title":"A random Title 3","body":"3- a block of text that can contain any character\nand it  also can contain multiple lines"}]

Code:

(awk
  (:set rs "\n\n" fs "\n")
  (:let (vec (vec)))
  ((and (equal [f 1] "BLOCK")
        (equal [f -1] "BLOCK"))
   (vec-push vec #J^{"title":~[f 0], "body":~(cat-str [f 2..-1])})
   (next))
  (t (error "bad data"))
  (:end (put-jsonl vec)))

The (:set ...) block is for initializations, and we use that to set up the record separator rs and field separator fs, which are analogous to the original Awk RS and FS. With a field separator that is a newline and a record separator that is a double newline, we get each information block as a record whose fields look like this:

"title" "BLOCK" "body1" "body2" ... "bodyn" "BLOCK"

In the awk macro, the fields are available as the list named f.

The main logic is a (condition action) pair. The condition is:

(and (equal [f 1] "BLOCK") (equal [f -1] "BLOCK"))

which is true if the second element of f and the last element are the string "BLOCK". If that is true, the action is executed, which extracts the pieces and adds an item to vec with the help of a JSON quasiquote like in the first program. We also execute (next) to move to the next record to avoid hitting the next condition-action pair.

The next condition-action pair, (t (error ...)) always executes because t is true, and throws an exception.

We print the JSON in the (:end ..) block, which is like END { ... } in classic Awk.

Speaking of error checking, the first program tolerates bad data to some extent; there are ways to fine tune it to reject bad inputs. For instance there can be junk between the records which is silently skipped, and if the last closing BLOCK is missing, that's okay.

1

Using any awk in any shell on every Unix box:

$ cat tst.awk
BEGIN {
    RS = ""
    FS = "\n"
    printf "["
}
{
    gsub(/"/,"\\\\&")

    title = $1
    body  = $3
    for (i=4; i<NF; i++) {
        body = body "\\n" $i
    }

    print  (NR>1 ? "," : "")
    print  "    {"
    printf "        \"title\": \"%s\",\n", title
    printf "        \"body\": \"%s\"\n",   body
    printf "    }"
}
END {
    print "\n]"
}

$ awk -f tst.awk file
[
    {
        "title": "A random Title 1",
        "body": "1- a block of text that can contain any character\nand it  also can contain multiple lines"
    },
    {
        "title": "A random Title 2",
        "body": "2- a block of text that can contain any character\nand it  also can contain multiple lines"
    },
    {
        "title": "A random Title 3",
        "body": "3- a block of text that can contain any character\nand it  also can contain multiple lines"
    }
]
1

We can use the Data::Dumper module to get a json output.

perl -MData::Dumper -00ne '
  my($title, $body) = 
    /^(.*?)\nBLOCK\n(.*)\nBLOCK\n$/s;

  push @A, {
    title => $title,
    body => $body,
  };

  END {
    $Data::Dumper::Useqq = 1;
    $Data::Dumper::Pair  = ": ";
    print Dumper \@A;
  }
' file | sed -Ee '1s/.{8}//;s/^ {8}//;$s/;$//'

Output:

[
  {
    "body": "1- a block of text that can contain any character\nand it  also can contain multiple lines",
    "title": "A random Title 1"
  },
  {
    "title": "A random Title 2",
    "body": "2- a block of text that can contain any character\nand it  also can contain multiple lines"
  },
  {
    "body": "3- a block of text that can contain any character\nand it  also can contain multiple lines",
    "title": "A random Title 3"
  }
]
1

Python along with the itertools module to group the input into chunks/paragraphs and then print them in a json style using the json module with the dumps method.

python3 -c 'import sys, json, itertools
ifile,rs = sys.argv[1],chr(10)

lod = []
with open(ifile) as fh:
  for k,g in itertools.groupby(fh, lambda x: x == rs):
    if not k:
      para = list(g)
      title,x,*body = list(map(lambda x: x.rstrip(rs),para[0:-1]))
      lod.append({
        "title": title,
        "body": rs.join(body)
      })

print(json.dumps(lod, sort_keys=False, indent=4))
' file

Output:

[
    {
        "title": "A random Title 1",
        "body": "1- a block of text that can contain any character\nand it  also can contain multiple lines"
    },
    {
        "title": "A random Title 2",
        "body": "2- a block of text that can contain any character\nand it  also can contain multiple lines"
    },
    {
        "title": "A random Title 3",
        "body": "3- a block of text that can contain any character\nand it  also can contain multiple lines"
    }
]

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .