awk/sed/perl one liner + how to print only the properties lines from json file

Question

how to print only the properties lines from json file

example of json file

{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]

expected output

    "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
    "is_supported_kafka_ranger" : "true",
    "kafka_log_dir" : "/var/log/kafka",
    "kafka_pid_dir" : "/var/run/kafka",
    "kafka_user" : "kafka",
    "kafka_user_nofile_limit" : "128000",
    "kafka_user_nproc_limit" : "65536"

Related question on SO: Parsing JSON with Unix tools
– hoefling
May 27, 2018 at 15:59 — hoefling, May 27, 2018 at 15:59
Also related: stackoverflow.com/questions/1732348/…
– Kartik
May 28, 2018 at 7:52 — Kartik, May 28, 2018 at 7:52

RomanPerekhrest · Accepted Answer · 2018-05-30 11:41:57Z

Jq is the right tool for processing JSON data:

jq '.items[].properties | to_entries[] | "\(.key) : \(.value)"' input.json

The output:

"content : \n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi"
"is_supported_kafka_ranger : true"
"kafka_log_dir : /var/log/kafka"
"kafka_pid_dir : /var/run/kafka"
"kafka_user : kafka"
"kafka_user_nofile_limit : 128000"
"kafka_user_nproc_limit : 65536"

In case if it's really mandatory to obtain each key and value double-quoted - use the following modification:

jq -r '.items[].properties | to_entries[]
       | "\"\(.key)\" : \"\(.value | gsub("\n";"\\n"))\","' input.json

The output:

"content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e "/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
"is_supported_kafka_ranger" : "true",
"kafka_log_dir" : "/var/log/kafka",
"kafka_pid_dir" : "/var/run/kafka",
"kafka_user" : "kafka",
"kafka_user_nofile_limit" : "128000",
"kafka_user_nproc_limit" : "65536",

You advocate using a syntax-aware tool (jq) rather than naive string operations, which is good, but then you use a naive string operation to do (limited) escape sequence processing for output. That doesn't seem like a good idea to me. jq must have a way to properly escape the value for output, right? — Daniel Pryden, May 28, 2018 at 19:05
@DanielPryden, No. Although jq does have some ways to properly escape the value for output (like @text, @sh etc), those won't help in this particular case. — RomanPerekhrest, May 28, 2018 at 19:47
A variant that leaves the properties values as JSON objects and uses sed to strip the unwanted braces and whitespace: jq '.items[].properties' input.json | sed -n 's/^\s\+//p' — Joe Lee-Moyet, May 29, 2018 at 12:54
why "," isn't appears in the output , as my expected results ? — yael, May 30, 2018 at 9:56
can you see please my "expected output" , can you edit your answer according to my expected results ? — yael, May 30, 2018 at 10:03

Stephen Kitt · Accepted Answer · 2018-05-27 17:50:10Z

Please, please don’t get into the habit of parsing structured data with unstructured tools. If you’re parsing XML, JSON, YAML etc., use a specific parser, at least to convert the structured data into a more appropriate form for AWK, sed, grep etc.

In this case, gron would help greatly:

$ gron yourfile | grep -F .properties.
json.items[0].properties.content = "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=/usr/lib/ccache:/home/steve/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi";
json.items[0].properties.is_supported_kafka_ranger = "true";
json.items[0].properties.kafka_log_dir = "/var/log/kafka";
json.items[0].properties.kafka_pid_dir = "/var/run/kafka";
json.items[0].properties.kafka_user = "kafka";
json.items[0].properties.kafka_user_nofile_limit = "128000";
json.items[0].properties.kafka_user_nproc_limit = "65536";

(You can post-process this with | cut -d. -f4- | gron --ungron to get something very close to your desired output, albeit still as valid JSON.)

jq is also appropriate.

nohillside · Accepted Answer · 2018-05-27 15:25:40Z

1

From Sed - An Introduction and Tutorial by Bruce Barnett:

sed -n '/properties/,/}$/ {
            /properties/n
            /}$/ !p
        }' FILE.json

For a more exact match and to also take care of closing bracket lines with additional whitespace you can use

sed -E -n '/"properties" : {/,/^[[:blank:]]*}[[:blank:]]$/ {
               /"properties" : {/n
               /^[[:blank:]]*}[[:blank:]]$/ !p
           }' FILE.json

edited May 27, 2018 at 15:25

answered May 27, 2018 at 14:53

nohillside

3,24117 silver badges27 bronze badges

I am not familiar with JSON but maybe /}/ is safer than /}$. The latter seems not to have any advantages anyway.
– Hauke Laging
May 27, 2018 at 15:05
1

@HaukeLaging Without the end-of-line marker it already matches the content line which contains a } somewhere.
– nohillside
May 27, 2018 at 15:09
5

Even though it is possible, it will most likely only work on the example file. If you want to parse structured data, you should rather use something designed for that. Be it jq, xpath, yq, xq, etc. That's because parsing it with line-oriented tools will eventually bite you in the back and debugging that might not be very easy.
– nert
May 27, 2018 at 17:20
For example, what happens if one of the 'href' fields contains the word "properties"?
– Stig Hemmer
May 28, 2018 at 8:45
1

@StigHemmer That's why I extended the pattern in the second example. But I totally agree that using gron or jq is the better approach.
– nohillside
May 28, 2018 at 10:48

Add a comment |

steve · Accepted Answer · 2018-05-28 08:44:28Z

1

sed one liner. Print lines between regular expression properties (i.e line containing "properties") and regular expression ^ *} (i.e. line starting with zero or more spaces followed by "}" and end-of-line).

sed -n '/properties/,/^ *}$/{//!p}' file.json

awk one liner.

awk '/^ *}/{s=0}/properties/{getline;s=1}s' file.json

edited May 28, 2018 at 8:44

answered May 27, 2018 at 22:14

steve

21.7k5 gold badges48 silver badges75 bronze badges

maybe you could explain how does your pattern matching works.
– vfbsilva
May 27, 2018 at 22:32
1

While this works for the example file given, it is risky trying to parse JSON with tools that doesn't understand it. For example, what happens if one of the 'href' fields contains the word "properties"? It is far less bug-prone to a JSON-aware tool like the top-voted answers.
– Stig Hemmer
May 28, 2018 at 8:42
3

Agree, risky. But OP specifically wanted a one-liner solution using sed/awk/perl. The answer I've given meet all these criteria.
– steve
May 28, 2018 at 8:46
What does //!p mean? Print if not one of the things that matched?
– David Conrad
May 29, 2018 at 0:29
1

Ah, got it, // repeats the last regex, ! not, p print. NIce.
– David Conrad
May 29, 2018 at 0:38

Add a comment |

Sobrique · Accepted Answer · 2018-05-31 10:45:40Z

It's tagged perl, and I see no perl answer yet, so I'll chip in.

Don't use regular expressions or other 'unstructured' parsers. perl has the JSON module with it. (JSON::PP is part of the core since 5.14 too)

#!/usr/bin/env perl

use strict;
use warnings;
use JSON;
use Data::Dumper;

my $str = do { local $/; <DATA> };

my $json = decode_json ( $str );

my $properties = $json -> {items} -> [0] -> {properties}; 

#dump the whole lot:
print Dumper $properties;


# or iterate
foreach my $key ( sort keys %$properties ) { 
   print "$key => ", $properties -> {$key},"\n";
}


__DATA__
{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]
}

Naturally you'd read from STDIN or a filename rather than DATA in your real usage scenario.

Stack Exchange Network

awk/sed/perl one liner + how to print only the properties lines from json file

5 Answers 5

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
awk
sed
perl
json
jq
.

Hot Network Questions

awk/sed/perl one liner + how to print only the properties lines from json file

5 Answers 5

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged awksedperljsonjq.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
awk
sed
perl
json
jq
.