Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Where I work, we have a site-to-site VPN tunnel set up between our main data center and a third party data center that does some Oracle PaaS for us. Since they charge per-VM, we're monitoring the Oracle audit logs on a local VM that is running the analytics for the Security and Audits department. The third party (or rather a department within it) are complaining about the amount of network traffic going over the tunnel due to syslog.

It's currently running unencrypted over udp/514, so I think I need to do some compression. After some research it looks like both rsyslog (what the third party is using) and syslog-ng (what we're using) support TLS compression. We have the CPU cycle's to spare, the third party's networking team is just complaining because apparently it impacts their SLA's.

My question is this:

Is there any way for rsyslog to essentially batch together several messages at once (say over a five minute window) and have them sent to our syslog-ng server all at once?

The reason I'm interested in doing this is because we would be switching over to TCP there's going to be more overhead. It's my understanding that most compression methods will do substitution on redundant data, which I'd imagine would be the case with a stream of syslog messages. The goal is to batch together then remove the redundant parts prior to transmission.

share|improve this question
    
TLS isn't compression, it's encryption. –  kurtm Oct 10 '13 at 14:15
    
@kurtm It has compression options –  Joel Davis Oct 10 '13 at 14:15
    
see also –  Joel Davis Oct 10 '13 at 14:16
    
I'd avoid using TCP for syslog. It's not only more overhead, it requires acknowledgement. So the clients sending to the syslog server will wait for acknowledgement, and start freezing until they get acknowledgements. I saw one department have their whole infrastructure freeze because their syslog server got bogged down. –  kurtm Oct 10 '13 at 14:17
    
I stand corrected on the compression. –  kurtm Oct 10 '13 at 14:17

1 Answer 1

up vote 1 down vote accepted

Sounds like rsyslog queueing might do what you want. Messages can also be stored for transmission during off-peak hours.

Specifically, the following:

The "$QueueDequeueSlowdown" directive allows to specify how long (in microseconds) dequeueing should be delayed.

share|improve this answer
    
Yep this is the answer. It looks like DequeueSlowdown does the exact thing I was asking for. I want to accept this as the answer but right now it's just a link to another website (technically answers are supposed to be self-contained). Can you edit your post to illustrate DequeueSlowdown ? –  Joel Davis Oct 11 '13 at 21:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.