2

Given the following output, how can I extract retention.ms's value of 3333?

Ultimately I'd like to print out topic has retention of 3333 milliseconds from the output of the kafka-topics.sh ... command:

$kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic topic
Topic: topic    cleanup.policy=delete,segment.bytes=999,retention.ms=3333,flush.messages=444

I shortened the text for brevity.

3
  • 1
    Is it OK to hard-code the word topic in the output, or do you need to capture the second word of the input?   Will the retention value always be presented as retention.ms= followed by an integer, or are things like retention.seconds=3.333 possible? Commented May 13 at 18:28
  • 1
    Can't you get the output as JSON and then parse that with jq? I believe the script has an option for producing output in other more easily parsable formats. Is it --output json?
    – Kusalananda
    Commented May 14 at 8:32
  • I did not find any --output json option in kafka-topics.sh. Commented May 21 at 22:16

3 Answers 3

5

I wouldn't use awk. If you've got GNU grep or Perl you could use a look-behind pattern:

k='Topic: topic    cleanup.policy=delete,segment.bytes=999,retention.ms=3333,flush.messages=444'
grep -oP 'retention.ms=\K\d+' <<<"$k"

Output

3333

Or sed

sed -nE 's/^.*\<retention.ms=([0-9]+).*/\1/p' <<<"$k"
3333

Or perl

perl -e '<> =~ /\bretention.ms=(\d+)/; print $1, "\n"' <<<"$k"
3333

They're all doing much the same thing: looking for retention.ms= and capturing the digits that follow.

2
  • 1
    Of course the sed command can be changed to 's/^.*\<retention.ms=([0-9]+).*/topic has retention of \1 milliseconds/p' to get the output asked for in the question.  (And you can leave off the ^.) Commented May 13 at 18:28
  • 2
    @G-ManSays'ReinstateMonica' indeed, but I answered the question in the first sentence rather than the second :) Commented May 13 at 21:13
4

Any time you have tag=value input I find it best to first create an array mapping the tags (names) to the associated values, i.e. f[] below, and then you can test, compare, print, select, or do anything else you want with your input just by indexing the array with whatever tag(s) you care about regardless of whatever order the input fields were in and whether any might be missing.

For example, using cat file instead of your kafka-topics command just for demo purposes and using any POSIX awk:

$ cat file |
    awk -F'[[:space:]:,=]+' '
        {
            delete f
            for ( i=1; i<NF; i+=2 ) {
                f[$i] = $(i+1)
            }
            printf "%s has retention of %d milliseconds\n", f["Topic"], f["retention.ms"]
        }
    '
topic has retention of 3333 milliseconds

If your real data has quoted fields containing spaces or anything then just change the loop populating f[] to suit, there's plenty of examples of such code online.

Here's a version of the above tweaked so you can see all the tag=value pairs that f[] contains from reading your input line:

cat file |
    awk -F'[[:space:]:,=]+' '
        {
            delete f
            for ( i=1; i<NF; i+=2 ) {
                f[$i] = $(i+1)
            }
            for ( tag in f ) {
                val = f[tag]
                printf "tag %s has value %s\n", tag, val
            }
        }
    '
tag flush.messages has value 444
tag segment.bytes has value 999
tag retention.ms has value 3333
tag cleanup.policy has value delete
tag Topic has value topic

so you can write code like:

if ( (f["flush.messages"] < 500) && (f["cleanup.policy"] == "delete") ) {
    if ( f["segment.bytes"] > f["flush.messages"] ) {
        print f["retention.ms"], f["segment.bytes"], f["Topic"]
    }
}

which demonstrates that you can compare whatever you like and print whatever you like in whatever order you like just using the tags as indices for f[], all you have to do is write the code to populate f[] and that's usually trivial to do as in my first script above.

0
1

Using Raku (formerly known as Perl_6)

~$ raku -ne '.split(/ <?after topic> \s+ /, 2) andthen   \
             say "retention.ms is ", %( .[1].split(/ <+[=,]> /)){"retention.ms"};'  file

OR:

~$ raku -ne 'BEGIN my ($topic, $key) = ("topic", "retention.ms"); 
             .split(/ <?after $topic> \s+ /, 2) andthen    \ 
             say "$key is ",  %( .[1].split(/ <+[=,]> /)){$key};'  file

Raku is a programming language in the Perl-family. Like Perl itself, Raku has hashes / maps (i.e. paired key/value storage) built-in.

  • We start by splitting the line into 2 elements using a positive lookbehind , which is spelled <?after X > Y and can be read "find Y after X".
  • The andthen conjunction explicitly reloads the topic variable, such that we can use .[0] and .[1] indexing in the 'clause' that follows.
  • In the first answer, the .[1] second-element-after-splitting is further split on <+[=,]> a custom-character class consisting of equals/comma. These elements are %(…) coerced to a hash, and the value specifically associated with "retention.ms" key is obtained with the familiar %hash{$key} idiom.
  • The second answer is identical to the first, except we use a BEGIN block to abstract out the desired $topic and $key. Again the value specifically associated with "retention.ms" key is obtained with the familiar %hash{$key} idiom.

Sample Input:

Topic: topic    cleanup.policy=delete,segment.bytes=999,retention.ms=3333,flush.messages=444

Sample Output:

retention.ms is 3333

Finally, as in the excellent awk answer by @EdMorton, you can simply loop through the hash to get a listified return of key/value pairs:

~$ printf 'Topic: topic    cleanup.policy=delete,segment.bytes=999,retention.ms=3333,flush.messages=444\n' |
   raku -ne '.split(/ <?after topic> \s+ /, 2) andthen  \
             say("\n" ~ .[0] ~ "\n----"), my %hash =  %( .[1].split(/ <+[=,]> /));
             say "tag {.key} has value {.value}" for %hash.sort: *.key;'

Topic: topic
----
tag cleanup.policy has value delete
tag flush.messages has value 444
tag retention.ms has value 3333
tag segment.bytes has value 999

https://docs.raku.org/language/regexes#Lookbehind_assertions
https://docs.raku.org/language/hashmap
https://raku.org

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .