10

how to print only the properties lines from json file

example of json file

{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]

expected output

    "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
    "is_supported_kafka_ranger" : "true",
    "kafka_log_dir" : "/var/log/kafka",
    "kafka_pid_dir" : "/var/run/kafka",
    "kafka_user" : "kafka",
    "kafka_user_nofile_limit" : "128000",
    "kafka_user_nproc_limit" : "65536"
33

Jq is the right tool for processing JSON data:

jq '.items[].properties | to_entries[] | "\(.key) : \(.value)"' input.json

The output:

"content : \n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi"
"is_supported_kafka_ranger : true"
"kafka_log_dir : /var/log/kafka"
"kafka_pid_dir : /var/run/kafka"
"kafka_user : kafka"
"kafka_user_nofile_limit : 128000"
"kafka_user_nproc_limit : 65536"

In case if it's really mandatory to obtain each key and value double-quoted - use the following modification:

jq -r '.items[].properties | to_entries[]
       | "\"\(.key)\" : \"\(.value | gsub("\n";"\\n"))\","' input.json

The output:

"content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e "/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
"is_supported_kafka_ranger" : "true",
"kafka_log_dir" : "/var/log/kafka",
"kafka_pid_dir" : "/var/run/kafka",
"kafka_user" : "kafka",
"kafka_user_nofile_limit" : "128000",
"kafka_user_nproc_limit" : "65536",
  • You advocate using a syntax-aware tool (jq) rather than naive string operations, which is good, but then you use a naive string operation to do (limited) escape sequence processing for output. That doesn't seem like a good idea to me. jq must have a way to properly escape the value for output, right? – Daniel Pryden May 28 '18 at 19:05
  • @DanielPryden, No. Although jq does have some ways to properly escape the value for output (like @text, @sh etc), those won't help in this particular case. – RomanPerekhrest May 28 '18 at 19:47
  • A variant that leaves the properties values as JSON objects and uses sed to strip the unwanted braces and whitespace: jq '.items[].properties' input.json | sed -n 's/^\s\+//p' – Joe Lee-Moyet May 29 '18 at 12:54
  • why "," isn't appears in the output , as my expected results ? – yael May 30 '18 at 9:56
  • can you see please my "expected output" , can you edit your answer according to my expected results ? – yael May 30 '18 at 10:03
26

Please, please don’t get into the habit of parsing structured data with unstructured tools. If you’re parsing XML, JSON, YAML etc., use a specific parser, at least to convert the structured data into a more appropriate form for AWK, sed, grep etc.

In this case, gron would help greatly:

$ gron yourfile | grep -F .properties.
json.items[0].properties.content = "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=/usr/lib/ccache:/home/steve/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi";
json.items[0].properties.is_supported_kafka_ranger = "true";
json.items[0].properties.kafka_log_dir = "/var/log/kafka";
json.items[0].properties.kafka_pid_dir = "/var/run/kafka";
json.items[0].properties.kafka_user = "kafka";
json.items[0].properties.kafka_user_nofile_limit = "128000";
json.items[0].properties.kafka_user_nproc_limit = "65536";

(You can post-process this with | cut -d. -f4- | gron --ungron to get something very close to your desired output, albeit still as valid JSON.)

jq is also appropriate.

2

From Sed - An Introduction and Tutorial by Bruce Barnett:

sed -n '/properties/,/}$/ {
            /properties/n
            /}$/ !p
        }' FILE.json

For a more exact match and to also take care of closing bracket lines with additional whitespace you can use

sed -E -n '/"properties" : {/,/^[[:blank:]]*}[[:blank:]]$/ {
               /"properties" : {/n
               /^[[:blank:]]*}[[:blank:]]$/ !p
           }' FILE.json
  • I am not familiar with JSON but maybe /}/ is safer than /}$. The latter seems not to have any advantages anyway. – Hauke Laging May 27 '18 at 15:05
  • 1
    @HaukeLaging Without the end-of-line marker it already matches the content line which contains a } somewhere. – nohillside May 27 '18 at 15:09
  • 5
    Even though it is possible, it will most likely only work on the example file. If you want to parse structured data, you should rather use something designed for that. Be it jq, xpath, yq, xq, etc. That's because parsing it with line-oriented tools will eventually bite you in the back and debugging that might not be very easy. – nert May 27 '18 at 17:20
  • For example, what happens if one of the 'href' fields contains the word "properties"? – Stig Hemmer May 28 '18 at 8:45
  • 1
    @StigHemmer That's why I extended the pattern in the second example. But I totally agree that using gron or jq is the better approach. – nohillside May 28 '18 at 10:48
2

sed one liner. Print lines between regular expression properties (i.e line containing "properties") and regular expression ^ *} (i.e. line starting with zero or more spaces followed by "}" and end-of-line).

sed -n '/properties/,/^ *}$/{//!p}' file.json

awk one liner.

awk '/^ *}/{s=0}/properties/{getline;s=1}s' file.json
  • maybe you could explain how does your pattern matching works. – vfbsilva May 27 '18 at 22:32
  • 1
    While this works for the example file given, it is risky trying to parse JSON with tools that doesn't understand it. For example, what happens if one of the 'href' fields contains the word "properties"? It is far less bug-prone to a JSON-aware tool like the top-voted answers. – Stig Hemmer May 28 '18 at 8:42
  • 3
    Agree, risky. But OP specifically wanted a one-liner solution using sed/awk/perl. The answer I've given meet all these criteria. – steve May 28 '18 at 8:46
  • What does //!p mean? Print if not one of the things that matched? – David Conrad May 29 '18 at 0:29
  • 1
    Ah, got it, // repeats the last regex, ! not, p print. NIce. – David Conrad May 29 '18 at 0:38
0

It's tagged perl, and I see no perl answer yet, so I'll chip in.

Don't use regular expressions or other 'unstructured' parsers. perl has the JSON module with it. (JSON::PP is part of the core since 5.14 too)

#!/usr/bin/env perl

use strict;
use warnings;
use JSON;
use Data::Dumper;

my $str = do { local $/; <DATA> };

my $json = decode_json ( $str );

my $properties = $json -> {items} -> [0] -> {properties}; 

#dump the whole lot:
print Dumper $properties;


# or iterate
foreach my $key ( sort keys %$properties ) { 
   print "$key => ", $properties -> {$key},"\n";
}


__DATA__
{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]
}

Naturally you'd read from STDIN or a filename rather than DATA in your real usage scenario.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.