Where I work, we have a site-to-site VPN tunnel set up between our main data center and a third party data center that does some Oracle PaaS for us. Since they charge per-VM, we're monitoring the Oracle audit logs on a local VM that is running the analytics for the Security and Audits department. The third party (or rather a department within it) are complaining about the amount of network traffic going over the tunnel due to syslog.

It's currently running unencrypted over udp/514, so I think I need to do some compression. After some research it looks like both rsyslog (what the third party is using) and syslog-ng (what we're using) support TLS compression. We have the CPU cycle's to spare, the third party's networking team is just complaining because apparently it impacts their SLA's.

My question is this:

Is there any way for rsyslog to essentially batch together several messages at once (say over a five minute window) and have them sent to our syslog-ng server all at once?

The reason I'm interested in doing this is because we would be switching over to TCP there's going to be more overhead. It's my understanding that most compression methods will do substitution on redundant data, which I'd imagine would be the case with a stream of syslog messages. The goal is to batch together then remove the redundant parts prior to transmission.

  • TLS isn't compression, it's encryption. – kurtm Oct 10 '13 at 14:15
  • @kurtm It has compression options – Bratchley Oct 10 '13 at 14:15
  • see also – Bratchley Oct 10 '13 at 14:16
  • I'd avoid using TCP for syslog. It's not only more overhead, it requires acknowledgement. So the clients sending to the syslog server will wait for acknowledgement, and start freezing until they get acknowledgements. I saw one department have their whole infrastructure freeze because their syslog server got bogged down. – kurtm Oct 10 '13 at 14:17
  • I stand corrected on the compression. – kurtm Oct 10 '13 at 14:17

Sounds like rsyslog queueing might do what you want. Messages can also be stored for transmission during off-peak hours.

Specifically, the following:

The "$QueueDequeueSlowdown" directive allows to specify how long (in microseconds) dequeueing should be delayed.

| improve this answer | |
  • Yep this is the answer. It looks like DequeueSlowdown does the exact thing I was asking for. I want to accept this as the answer but right now it's just a link to another website (technically answers are supposed to be self-contained). Can you edit your post to illustrate DequeueSlowdown ? – Bratchley Oct 11 '13 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.