Where I work, we have a site-to-site VPN tunnel set up between our main data center and a third party data center that does some Oracle PaaS for us. Since they charge per-VM, we're monitoring the Oracle audit logs on a local VM that is running the analytics for the Security and Audits department. The third party (or rather a department within it) are complaining about the amount of network traffic going over the tunnel due to syslog.
It's currently running unencrypted over udp/514, so I think I need to do some compression. After some research it looks like both
rsyslog (what the third party is using) and
syslog-ng (what we're using) support TLS compression. We have the CPU cycle's to spare, the third party's networking team is just complaining because apparently it impacts their SLA's.
My question is this:
Is there any way for
rsyslog to essentially batch together several messages at once (say over a five minute window) and have them sent to our
syslog-ng server all at once?
The reason I'm interested in doing this is because we would be switching over to TCP there's going to be more overhead. It's my understanding that most compression methods will do substitution on redundant data, which I'd imagine would be the case with a stream of syslog messages. The goal is to batch together then remove the redundant parts prior to transmission.