Following up on the heels of this question, https://stackoverflow.com/questions/40768603/logstash-trying-to-make-sense-of-strings-passed-by-intelmq-in-elasticsearch I am trying to create a refine/create a filter to receive messages from logstash to kibana.

Whilst the original requirements and answer are almost to speck, some new bots added to IntelMQ now put spaces on the fields. Obviously, they break completely the filters, and worse yet create spurious new fields and date in Elastic Search.

I also have found out the solution in the referred thread does take well in account the beginning and end of strings.

The strings itself is similar to:

{u'feed': u'openbl', u'reported_source_ip': u'', u'source_cymru_cc': u'VN', u'source_time': u'2016-06-25T11:15:14+00:00', u'feed_url': u'http://www.openbl.org/lists/date_all.txt', u'taxonomy': u'Other', u'observation_time': u'2016-11-20T22:51:25', u'source_ip': u'', u'source_registry': u'apnic', u'source_allocated': u'2008-07-17', u'source_bgp_prefix': u'', u'type': u'blacklist', u'source_as_name': u'VIETEL-AS-AP Viettel Corporation, VN', u'source_asn':u'7552'}

What to do?


The obvious solution was to ask logstash to enclose the record of each field into '[]with the optioninclude_brackets => true`, and then adding filters to handle that.

At the end of the day, the correct filter to import data taking in account a string coming from redis, in /etc/logstash/conf.d/filter.conf is:

filter {
  mutate {
    gsub => [
       "message", "{u'", "",
       "message", "': u'", ": [",
       "message", "', u'", "], ",
       "message", "'}", "]"
  kv {
    source => "message"
    field_split => ", "
    value_split => ": "
    remove_tag => [ "_jsonparsefailure" ]
    include_brackets => true

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.