1

Running CentOS 7.7 in VPS

Question:
Is there native functionality to copy|duplicate|sync lines from a specific log file matching specific search criterion to a separate data file for archival reference?

Purpose:
I'm running CSF firewall and wish to archive log lines indicating a block for the purpose of periodic statistical analysis of prevalent network offenders based on ASN.

The firewall allows blocking by country or ASN which has been a very effective method for reducing spam and port scanners, eliminating 96+% of undesired traffic. Previously, I have copied the transformed data to Google Sheets for analysis via pivotable.

Log Example:

/var/log/lfd.log
Sep 21 06:14:30 server lfd[7330]: *Port Scan* detected from 120.202.192.171 (CN/China/-/-/-/[AS9808 Guangdong Mobile Communication Co.Ltd.]). 6 hits in the last 230 seconds - *Blocked in csf* for 36000 secs [PS_LIMIT]

Notes:
• Matching lines are easily identified by 'Blocked in csf' and '[AS'.
• The log can receive upwards of 120 of these entries per day at any moment.
• This log file is rotated every Sunday morning some time after 0300; the starting and ending times are inconsistent; entries are commonly entered during this period.
• The rotated file is gzipped and renamed: /var/log/lfd.log-YYYYMMDD.gz
• The date is in an irregular format for statistical analysis and about half of the log line is not needed.

Caveats:
I'm presuming a real-time solution is ideal since this will avoid the need to keep track of which entries have been copied and which haven't. Otherwise, the solution will need to address the issue of the log being rotated and compressed periodically at an inconsistent time.

Ideally, I'd like to also transform the data into a regular, delimited format as it is copied to the archive file to examine the most prevalent (frequency) offenders by date period. I only need in CSV or TSV format:

[DATE], [COUNTRY], [ASN], [ISP]

I'd like to analyze these offenders on a daily to weekly basis so looking to streamline this as much as possible, including simple statistical analysis over selective date periods via the command line. I have no experience with stats on linux so also looking for a recommendation on native stats package that's easy to invoke on the archive file. So far, PSPP looks like the most effective tool for this job.

Thanks in advance for your advice on creating an archive of selective log lines, possibly transforming them in the process, and maybe a native stats package.

  • Regarding the selection & processing of log messages, rsyslog might be a good fit. It can read input files, apply filter conditions on messages, send the messages to different output files and even has native support for the CSV format. – Haxiel Sep 22 '19 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.