Currently we do not have any log tools for filtering a huge dumb file. Is there any open source BI or log tool able to import a dumb file and do auto/selectable filtering?

Current dumb file example:

2019-02-11 03:06:30.202  INFO maskParams=false
2019-02-11 03:06:30.205  INFO Start request :: ipAddress=
2019-02-11 03:06:30.205  INFO  User-Agent = Mozilla/5.0 (Linux; U; Android 7.1.1; zh-cn; MI MAX 2 Build/NMF26F) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.128 Mobile 
Safari/537.36 XiaoMi/MiuiBrowser/10.5.1
2019-02-11 03:06:30.205  INFO  Request Query String=[customer_name]
2019-02-11 03:06:30.335  INFO  Checking current ip address
2019-02-11 03:06:30.335  ERROR Not a valid ip address 2408:84e4:412:4b02:73cd:9684:1ab0:8437
2019-02-11 03:06:30.336  INFO  Get country from GeoIpRange, ip =2408:84e4:412:4b02:73cd:9684:1ab0:8437
2019-02-11 03:06:30.336  ERROR Not a valid ip address 2408:84e4:412:4b02:73cd:9684:1ab0:8437
2019-02-11 03:06:30.385  INFO  End request (00:00:00.183) 

After importing the dumb file, able to provide me the below details:

  • Start Date and time/
  • End Date and time/
  • customer name/
  • no valid ip address details

There are many open source log collectors and indexers:

Splunk? Elasticsearch? Logstash? Kibana?

Are they BI tools? I'm not sure. Only if you plot some business metrics I guess.

| improve this answer | |
  • i had tested Splunk and it really give me what I wanted. Thanks! – julian lee Feb 15 '19 at 3:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.