I am trying to configure a server that will have two syslog servers. One is storing all messages and the other is analyzing them for statistics.

The only way I can figure out how to do this is to have some sort of port mirror happening that looks for all packets inbound on port 514 (but still allows the original packet to go to rsyslog on port 514), copies them, then sends them to another port on the same box while preserving the source IP.

I found samplicator, but that wont work with a port that is already bound on the system.

How can I do this?

1 Answer 1


RSyslog Forwarding Setup


Set up TCP listening on the log index host Uncomment the following lines in /etc/rsyslog.conf. This will enable the rsyslog daemon to listen for incoming requests on TCP port 514. We're using TCP here so that we can have some confidence that the messages from the agent hosts reach the indexer. (More on this below)

Provides TCP syslog reception

$ModLoad imtcp
$InputTCPServerRun 514
Add a line to /etc/rsyslog.conf to actually put the received logs in a specific file.

local3.*                        /local/logs/httpd-error
local4.*                        /local/logs/httpd-access```
Finally, restart the rsyslog process.

```service rsyslog restart```

**Set up the agent host**

On the agent host, the host that is running apache, add a file, /etc/rsyslog.d/apache.conf. This will be read at syslog start time. This file tells rsyslog to read /var/log/httpd/error_log (the default apache error log on CentOS) every 10 seconds and send its messages to the local3.info facility in syslog. (Expanded to also read Access logs and send those to local4.info)

$ModLoad imfile

# Default Apache Error Log
$InputFileName /var/log/httpd/error_log
$InputFileTag httpd-error-default:
$InputFileStateFile stat-httpd-error
$InputFileSeverity info
$InputFileFacility local3

# Default Apache Access Log
$InputFileName /var/log/httpd/access_log
$InputFileTag httpd-access-default:
$InputFileStateFile stat-httpd-access
$InputFileSeverity info
$InputFileFacility local4

$InputFilePollInterval 10

Next, modify /etc/rsyslog.conf uncomment or add the following lines to the end of the file. This tells rsyslog to set up a log queue and forward any local3 and local4 facility log messages to the TCP port of @@ is rsyslog shorthand for the TCP syslog port. If you want to forward via UDP, use a single @ instead. However, it's probably not worth setting up the queue in that case, as rsyslog doesn't have a way to assure that UDP packets are received by the index host.

In this configuration the agent host will store any log messages that can't be sent to the index host. This is good for dealing with times that the index host is being rebooted, or otherwise unavailable.

$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
local3.* @@
local4.* @@

Lastly, restart rsyslog on the agent host.

service rsyslog restart

Test the configuration

Now, if you restart the apache server (service httpd restart) you should see logs being generated in /local/logs/httpd-error on the index host. If not check to see if there are any firewall blocks between the hosts, and that your rsyslog config changes are being parsed properly. You can check your rsyslog configuration with this command: /sbin/rsyslogd -c5 -f /etc/rsyslog.conf -N1

Configure Logstash to consume the new files

This configuration uses a custom grok parser to pull the error level out of the message, as well as pull the tag we set in the rsyslog config into its own field as well. For help with custom grok filters check out http://grokdebug.herokuapp.com/

  input {
  file {
    type => "httpd-error-log"
    path => ["/local/logs/httpd-error"]
    sincedb_path => "/opt/logstash/sincedb-access"
    discover_interval => 10

  file {
    type => "httpd-access-log"
    path => ["/local/logs/httpd-access"]
    sincedb_path => "/opt/logstash/sincedb-access"
    discover_interval => 10

filter {
  if [type] == "httpd-error-log" {
    grok {
      match       => [ "message", "\S+ \d+ \d+:\d+:\d+ %{HOSTNAME} %{NOTSPACE:tag}: \[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\] \[%{LOGLEVEL:level}\] %{GREEDYDATA}" ]
    mutate {
      rename      => [ "program", "vhost" ]

  if [type] == "httpd-access-log" {
    grok {
      match       => [ "message", "\S+ \d+ \d+:\d+:\d+ %{HOSTNAME} %{NOTSPACE:tag}: %{COMBINEDAPACHELOG}" ]
      add_field     => { "level", "info" }

output {
  elasticsearch {
    host => "localhost"

Check out this GitHub Project https://gist.github.com/GaryRogers/85c8f2a805010ceeccc6

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.