Strip newline characters and quotes from JSON array and set specific strings as variables via bash

Question

I have the following JSON data:

{
    "Name": "No.reply",
    "Email": "[email protected]",
    "ID": 5930,
    "details": 
    {
        "message": "Your name: john doe\nEmail: [email protected]\nSubject: I need help with this\nDescription: I can find the download for the manual but I can only find the free updater for Windows or Mac. Can you help me please as I have a chrome notebook and Moto smart phone. Thank you. John doe"
    }
}

The name and email fields from the top level are irrelevant, as they are from the automated email. The information I need is in the message field and in the ID field, which is related to John Doe's info.

Anyway, this is what I need to be filtered and how it should be saved to a new file in this order:

• Name: it should read the lines after this variable, regardless of the text.
• Email: same as above
• Subject: same as above
• Description: same as above
• ID: same as above

So, I need to remove the quotes, the newline character, assign those specific strings to a variable via bash, and read what it's after those strings.

I was able to come up with something, but it doesn't work for this JSON output: (only works if the text file is properly formatted)

while IFS=''
do  case "$line" in
    "Name:"*)             uservar="${line#*: }" ;;
    "Email:"*)            emailvar="${line#*: }" ;;
    "Subject:"*)          subject="${line#*: }" ;;
    "Message:"*)          message="${line#*: }" ;;
    "ID:"*)        ticketidvar="${line#*: }" ;;
    esac
done <<-EOF
$(pbpaste)
EOF

Answer 1

This assumes that the Description: ... part of the message is a single line, and that the headers are in the canonical form (no " subJECT :hey", please).

It's using jq's @sh format spec to escape its output in a manner suitable for the shell (with single quotes). Thanks to @Stéphane Chazelas for corrections.

parse_json(){
  jq=$(jq -r '[.Email,.ID,(.details.message | split("\n")) | @sh] | join(" ")' -- "$1")
  eval "set -- $jq"
  email=$1; shift
  id=$1; shift
  for l; do
    case $l in
    "Your name: "*) name="${l#*: }";;
    "Subject: "*) subject="${l#*: }";;
    "Description: "*) description="${l#*: }";;
    # remove the following line if you want the .Email from json instead
    "Email: "*) email="${l#*: }";;
    esac
  done
  echo "id={$id}"
  echo "name={$name}"
  echo "email={$email}"
  echo "subject={$subject}"
  echo "description={$description}"
}

fz:/tmp% parse_json a.json
id={5930}
name={john doe}
email={[email protected]}
subject={I need help with this}
description={I can find the download for the manual but I can only find the free updater for Windows or Mac. Can you help me please as I have a chrome notebook and Moto smart phone. Thank you. John doe

The case ... esac above could be replaced with something that will create variables with the same names as the headers with the non-alphanumeric characters replaced by underscores. This will only work with shells that support ${var//pat/repl} substitutions(bash, zsh, ksh93):

parse_json(){
  jq=$(jq -r '[.Email,.ID,(.details.message | split("\n")) | @sh] | join(" ")' -- "$1")
  eval "set -- $jq"
  Email=$1; shift
  ID=$1; shift
  for l; do
    v="${l#*: }"; k="${l%%: *}"; eval "${k//[!a-zA-Z0-9]/_}=\$v"
  done
}

show_vars(){
  for v in ID Your_name Email Subject Description; do
    eval "echo \"$v=[\$$v]\""
  done
}

fz:/tmp$ parse_json a.json
fz:/tmp$ show_vars
ID=[5930]
Your_name=[john doe]
Email=[[email protected]]
Subject=[I need help with this]
Description=[I can find the download for the manual but I can only find the free updater for Windows or Mac. Can you help me please as I have a chrome notebook and Moto smart phone. Thank you. John doe]

Answer 2

You could use an associative array:

typeset -A field="($(jq -r '
   "[ID]=" + (.ID|@sh),
   (.details.message|capture("(?m)^(?<t>.*?): (?<v>.*)"; "g")|
     "["+(.t|@sh)+"]="+(.v|@sh))' file.json))"

Then you'd have the ID in ${field[ID]}, subject in ${field[Subject]}...

On your sample, typeset -p field outputs:

declare -A fields=(
  [Description]="I can find the download for the manual but I can only find the free updater for Windows or Mac. Can you help me please as I have a chrome notebook and Moto smart phone. Thank you. John doe"
  [ID]="5930"
  [Subject]="I need help with this"
  [Email]="[email protected]"
  ["Your name"]="john doe"
)

Answer 3

I am not sure I understand you correctly, but if the intention is to create variables out of the contents of .details.message I would suggest the following.

First extract .details.message in raw mode with jq:

jq -r '.details.message' in.json

Now parse this with sed to make it bash-parsable:

parse.sed

/^Your name: /   s//uservar="/
/^Email: /       s//emailvar="/
/^Subject: /     s//subject="/
/^Description: / s//message="/
s/$/"/

Now you can source this in your script, e.g.:

. <(jq -r '.details.message' in.json | sed -f parse.sed)
echo $uservar, $emailvar

Output:

john doe, [email protected]

Answer 4

Not so sophisticated (I suppose the shown text is in some file let us say json.txt or you can send the text to the first sed through a pipe):

sed s/'$'/:/ json.txt |\ ... add a : at the end of each line

sed s/'\\n'/:/g |\ ... change any \n to :

sed s/'"'//g |\ ... remove quots

awk 'BEGIN{RS="}";FS=":"}{if($7){print "Name: "$3"\nEmail: "$5"\nSubject: "$17"\nDescription: "$19"\nID: "$7}}'

In awk set the row separator to }, field separator to : and compose your output. There may left some ending commas, you can easily remove.

Answer 5

Assuming that all you need is the decoded .details.message string followed by a line with the ID:

jq -r '.details.message, "ID: \(.ID)"' file >outfile

This decodes the .details.message string and adds a separate line with the value of the top-level .ID key, prefixed by ID: .

For the given JSON document, this would generate the following output in the file outfile:

Your name: john doe
Email: [email protected]
Subject: I need help with this
Description: I can find the download for the manual but I can only find the free updater for Windows or Mac. Can you help me please as I have a chrome notebook and Moto smart phone. Thank you. John doe
ID: 5930