3

We have the following file with hostnames and host ip's ( long file with 90-100 machines per linux machine )

hosts.cluster.conf

  "href" : "http://localhost:8080/api/v1/hosts/worker02.sys87.com",
  "Hosts" : 
    "cluster_name" : "hdp",
    "host_name" : "worker02.sys87.com",
    "ip" : "23.67.32.65"


  "href" : "http://localhost:8080/api/v1/hosts/worker03.sys87.com",
  "Hosts" : 
    "cluster_name" : "hdp",
    "host_name" : "worker03.sys87.com",
    "ip" : "23.67.32.66"


  "href" : "http://localhost:8080/api/v1/hosts/worker04.sys87.com",
  "Hosts" : 
    "host_name" : "worker04.sys87.com",
    "ip" : "23.67.32.67"


  "href" : "http://localhost:8080/api/v1/hosts/worker05.sys87.com",
  "Hosts" : 
    "cluster_name" : "hdp",
    "host_name" : "worker05.sys87.com",
    "ip" : "23.67.32.68"

we want to print all host_name lines only if the upper line before include the "cluster_name" word

expected results

"host_name" : "worker02.sys87.com",

"host_name" : "worker03.sys87.com",

"host_name" : "worker05.sys87.com",
  • 2
    grep -B 1 'host_name' file | grep -A 1 'cluster_name' | grep 'host_name'? – Cyrus Jan 7 '18 at 12:27
  • Is this a subset of something that was originally in JSON? – Will Crawford Jan 8 '18 at 1:00
3

Short awk solution:

awk '/cluster_name/{ cl=NR }/host_name/ && NR-1==cl' hosts.cluster.conf
  • /cluster_name/{ cl=NR } - capturing the record number of "cluster_name" line
  • /host_name/ - on encountering "host_name" line
  • NR-1==cl - ensuring that the current "host_name" record number NR is next after "cluster_name" record number (presented by cl)

The output:

"host_name" : "worker02.sys87.com",
"host_name" : "worker03.sys87.com",
"host_name" : "worker05.sys87.com",

In case if host_name appears as the 1st line, though I doubt about that in real case, use the following version:

awk '/cluster_name/{ cl=NR }/host_name/ && cl && NR-1==cl' hosts.cluster.conf
  • This will print out even host_name on first line. – PesaThe Jan 7 '18 at 15:59
  • @PesaThe, see my update, though your assumption is far from realistic structure of hosts.cluster.conf – RomanPerekhrest Jan 7 '18 at 16:03
  • True, it was just a nitpick. Anyway, nice edit :) – PesaThe Jan 7 '18 at 16:13
5
sed '/host_name/!h;//!d;x;/cluster_name/!d;g' infile

will save each line that doesn't match host_name to the hold buffer and then delete it; for each remaining line it will exchange buffers and, if the pattern space doesn't contain cluster_name it will delete it, else it will get the original line back from the hold buffer and autoprint.

3

try

 awk '/cluster_name/ {p=1 ; next ;} 
 /host_name/ && p { print ; }
 {p=0}' 

This basically remember line with cluster_name. If host_name is found, without cluster_name before, it won't be printed.

Note that the whole awk code can be one-lined.

  • 1
    {p=0} will always reset the p flag to 0. You probably forgot to use: {p=1; next}. – PesaThe Jan 7 '18 at 13:26
  • @PesaThe your are right, I fixed it. – Archemar Jan 7 '18 at 14:58
1
#!/usr/bin/perl

$/='';

while(<>) {
 next unless (m/"cluster_name"/);
 chomp;

 foreach my $l (split /\n/) {
    print $l, "\n\n" if ($l =~ m/"host_name"/);
 };
};

In English: Reading the file a paragraph at a time ($/='';), skip paragraphs that don't contain the string "cluster_name". For those that do contain that string, iterate through each line of the paragraph and print lines containing "host_name", with two newlines after each instance.

Example output:

$ ./extract-hosts.pl hosts.cluster.conf 
    "host_name" : "worker02.sys87.com",

    "host_name" : "worker03.sys87.com",

    "host_name" : "worker05.sys87.com",
  • anonymous downvoter: so what's wrong with this answer? Leave a comment to explan the problem. Given that the OP included /perl as one of the tags and this answer is the ONLY one that exactly reproduces the OP's requested output, is it just that you don't like perl? – cas Jan 8 '18 at 7:37
1

Well we have sed and awk, now time for GNU grep!

cat infile | grep --after-context 1 cluster_name | grep host_name

Explanation

The first command cat reads out the data into the pipe for processing. You can replace this segment with any command that outputs your source text to stdout.

The second command finds any line with "cluster_name" in it, and prints it and the following line. The intermediate output is this:

"cluster_name" : "hdp",
"host_name" : "worker02.sys87.com",
--
"cluster_name" : "hdp",
"host_name" : "worker03.sys87.com",
--
"cluster_name" : "hdp",
"host_name" : "worker05.sys87.com",

Then the final segment only prints the contents of the lines with "host_name" in them. Thus the final output is:

"host_name" : "worker02.sys87.com",
"host_name" : "worker03.sys87.com",
"host_name" : "worker05.sys87.com",

Remarks

  1. Not every grep has the --before-context parameter. Make sure you are using GNU grep and you should be fine.
  2. If this is language like JSON, you would be better off to learn and use a language appropriate parser like jmespath or jq.
0

This snippet:

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }

pl " Input data file $FILE:"
cat $FILE

pl " Expected output:"
cat $E

pl " Results:"
cgrep -D -w '"href"' +w '"ip"' "cluster_name" $FILE |
grep '"host_name"'

produces:

-----
 Input data file data1:
  "href" : "http://localhost:8080/api/v1/hosts/worker02.sys87.com",
  "Hosts" : 
    "cluster_name" : "hdp",
    "host_name" : "worker02.sys87.com",
    "ip" : "23.67.32.65"


  "href" : "http://localhost:8080/api/v1/hosts/worker03.sys87.com",
  "Hosts" : 
    "cluster_name" : "hdp",
    "host_name" : "worker03.sys87.com",
    "ip" : "23.67.32.66"


  "href" : "http://localhost:8080/api/v1/hosts/worker04.sys87.com",
  "Hosts" : 
    "host_name" : "worker04.sys87.com",
    "ip" : "23.67.32.67"


  "href" : "http://localhost:8080/api/v1/hosts/worker05.sys87.com",
  "Hosts" : 
    "cluster_name" : "hdp",
    "host_name" : "worker05.sys87.com",
    "ip" : "23.67.32.68"


  "href" : "http://localhost:8080/api/v1/hosts/worker06.sys87.com",
  "Hosts" : 
    "host_name" : "worker06.sys87.com",
    "cluster_name" : "hdp",
    "ip" : "23.67.32.69"

-----
 Expected output:
"host_name" : "worker02.sys87.com",

"host_name" : "worker03.sys87.com",

"host_name" : "worker05.sys87.com",

-----
 Results:
    "host_name" : "worker02.sys87.com",
    "host_name" : "worker03.sys87.com",
    "host_name" : "worker05.sys87.com",
    "host_name" : "worker06.sys87.com",

It first extracts all stanzas that contain "cluster_name". It does so by extracting stanzas that are regions (or windows: -w, +w) "href" .. "ip". From this subset, then a normal grep extracts the desired lines matching "host_name".

This scheme allows the lines "cluster_name" and "host_name" to appear anywhere in the stanza, even in different relative positions, such as in the additional stanza for "worker06".

Done on a system like:

OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.9 (jessie) 
bash GNU bash 4.3.30
cgrep ATT cgrep 8.15

Some details for cgrep:

cgrep   shows context of matching patterns found in files (man)
Path    : ~/executable/cgrep
Version : 8.15
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Home    : http://sourceforge.net/projects/cgrep/ (doc)

Best wishes ... cheers, drl

0

Another straightforward sed:

 sed -n '/host_name/ { x; /cluster_name/ { x; p; x }; x }; h' file
0

Here is yet another sed solution, that will perhaps be easier to understand than the others:

sed -n '/cluster_name/{n;/host_name/p}'

Here we just look for a line matching cluster_name, and if the next line matches host_name, we print it.

0

Below sed onliner command does the same.Tested its works fine

sed -n '/cluster_name/,+1p' hosts.cluster.conf | sed -n '/host_name/p'

output

  "host_name" : "worker02.sys87.com",
    "host_name" : "worker03.sys87.com",
    "host_name" : "worker05.sys87.com",

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.