1

in our company we need aggregate services statistics

for some reason we decide to use rsyslogd to send applications statistics (json format) to elasticsearch system directly (this tutorial).

but we faced this issue:

when send json statistics to local rsyslogd by logger command , everything was ok. enter image description here

but

when application (java) send these statistics to rsyslogd (with backlog framework) json escaped by backslash

mmjsonparse error:

9448.407432204:main Q:Reg/w0  : Called action, logging to mmjsonparse
9448.407443744:main Q:Reg/w0  : Action 1 transitioned to state: itx
9448.407450424:main Q:Reg/w0  : entering actionCalldoAction(), state: itx, actionNbr 1
9448.407465385:main Q:Reg/w0  : mmjsonparse: no JSON cookie: '{"subject":                "Report","report_no":              2411,"report_time_from":       1479309445405,"report_time_until":      1479309448406,"report_time_duration":   3001,"upload_lessThan1M_count":      0,"upload_lessThan1M_size":       0,"upload_btw1Mto2M_count":      0,"upload_btw1Mto2M_size":       0,"upload_btw2Mto5M_count":      0,"upload_btw2Mto5M_size":       0,"upload_btw5Mto10M_count":      0,"upload_btw5Mto10M_size":       0,"upload_btw10Mto20M_count":      0,"upload_btw10Mto20M_size":       0,"upload_btw20Mto50M_count":      0,"upload_btw20Mto50M_size":       0,"upload_btw50Mto100M_count":      0,"upload_btw50Mto100M_size":       0,"upload_moreThan100M_count":      0,"upload_moreThan100M_size":       0,"upload_total_size":  0,"upload_total_count": 0,"thumb_count":  0,"thumb_time":   0,"download_lessThan1M_count":      0,"download_lessThan1M_size":       0,"download_btw1Mto2M_count":      0,"download_btw1Mto2M_size":       0,"download_btw2Mto5M_count":      0,"download_btw2Mto5M_size":       0,"download_btw5Mto10M_count":      0,"download_btw5Mto10M_size":       0,"download_btw10Mto20M_count":      0,"download_btw10Mto20M_size":       0,"download_btw20Mto50M_count":      0,"download_btw20Mto50M_size":       0,"download_btw50Mto100M_count":      0,"download_btw50Mto100M_size":       0,"download_moreThan100M_count":      0,"download_moreThan100M_size":       0,"download_total_size":  0,"download_total_count": 0,"cache_served_count": 0,"cache_served_size":  0,"cache_new_count":    0,"cache_new_size":     0}'
9448.407510073:main Q:Reg/w0  : Action 1 transitioned to state: rdy
9448.407517838:main Q:Reg/w0  :     PRIFILT 'local2.*'
9448.407527643:main Q:Reg/w0  :     pmask:  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X FF  X  X  X  X  X  X  X 
9448.407598847:main Q:Reg/w0  : PRIFILT condition result is 1
9448.407604631:main Q:Reg/w0  :     ACTION 2 [omelasticsearch:action(type="omelasticsearch" ...)]
9448.407617909:main Q:Reg/w0  : executing action 2
9448.407622728:main Q:Reg/w0  : Called action, logging to omelasticsearch
9448.407631995:main Q:Reg/w0  : action 3 queue: qqueueAdd: entry added, size now log 1, phys 1 entries

software details: (rsyslog 8.4.2-1+deb8u2 , rsyslog-elasticsearch )

syslog-config:

#load needed modules
#load needed modules
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
module(load="mmjsonparse") #for parsing CEE-enhanced syslog messages
module(load="omelasticsearch") #for indexing to Elasticsearch


#try to parse structured logs
local2.* action(type="mmjsonparse")

#define a template to print field "foo"
template(name="justFoo" type="list") {
    property(name="$!all-json")
}

#and now let's write the contents of field "foo" in a file
#action(type="omfile"
#    template="justFoo"
#    file="/tmp/foo")

local2.*    action(type="omelasticsearch"
       server="192.168.218.42"
       serverport="9200"
       template="justFoo"
       searchIndex="stats"
       searchType="stats"
       bulkmode="on"
       queue.type="linkedlist"
       queue.size="5000"
       queue.dequeuebatchsize="300"
       action.resumeretrycount="-1")
1

I've found the answer!

local2.* action(type="mmjsonparse")

would change to:

action(type="mmjsonparse" cookie="")

Details:

Action specific Configuration Directives:

cookie [string] defaults to “@cee:”

Permits to set the cookie that must be present in front of the JSON part of the message.

Most importantly, this can be set to the empty string (“”) in order to not require any cookie. In this case, leading spaces are permitted in front of the JSON. No non-whitespace characters are permitted after the JSON. If such is required, mmnormalize must be used.

Ref: http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmjsonparse.html

Most importantly, this can be set to the empty string (“”) in order to not require any cookie. In this case, leading spaces are permitted in front of the JSON. No non-whitespace characters are permitted after the JSON. If such is required, mmnormalize must be used.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.