0

Edit: the actual problem ended up being either the local0 vs local2 distinction and the rsyslog end of deal.

I'm building a kubernetes / openshift cluster, and on front of that, I need to set up a L4 LB for api-server and application services. The traffic is handled at tcp level even though the traffic itself is actually https. The reason for this setup is that the kubernetes/openshift owns the certificates and their keys even though the ip address that's pointed by the DNS for the apiserver / http apps wildcard record is on the haproxy host.

Load balancing works, i.e. the traffic gets where it should without problem, but haproxy keeps insisting to write all of its logs into /var/log/messages instead of under /var/log/haproxy/haproxy.log, despite me doing all I find to do to make it happen. As far as I can tell:

  • rsyslog is listening on udp port 514
  • rsyslog is instructed to write those things into haproxy files as instructed in the haproxy documentation and in many answers
  • haproxy is instructed to send its logs to 127.0.0.514 .

Host system is RHEL8, haproxy 1.8.15 and rsyslog 8.37 are installed from official repos.

This is the relevant part of my /etc/haproxy/haproxy.cfg:

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local0

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

.
.
.

# ---------------------------------------------------------------------
#  round robin balancing for OCP Ingress Insecure Port
# ---------------------------------------------------------------------
frontend ingress_insecure
    bind               *:80
    mode               tcp
    option  tcplog
    default_backend    ingress_insecure_backend

backend ingress_insecure_backend
    balance  roundrobin
    mode     tcp

        server worker1.cluster.example.com <ip address>:80 check
        server worker2.cluster.example.com <ip address>:80 check

This is my /etc/rsyslog.d/haproxy :

local0.* /var/log/haproxy/haproxy.log
& stop

These lines exist in /etc/rsyslog.conf:

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

This is the output of netstat -tulpn:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
--- clip ---          
udp        0      0 0.0.0.0:514             0.0.0.0:*                           8031/rsyslogd
udp6       0      0 :::514                  :::*                                15270/rsyslogd 
--- clip ---

This is the output of ls: (I created the directory by hand and gave it very permissive access mode to make sure it won't stop there)

# ls -ld /var/log/haproxy
drwxrwxrwx 2 root root 6 Dec 18 14:40 /var/log/haproxy
# ls -la /var/log/haproxy
total 12
drwxrwxrwx   2 root root    6 Dec 18 14:40 .
drwxr-xr-x. 11 root root 8192 Dec 18 14:50 ..

Yet, the haproxy log data still only appears into /var/log/messages instead of under /var/log/haproxy/. Any pointers where to look further?

0

Ok, it seems that this https://linuxconfig.org/install-and-configure-haproxy-on-redhat-8 solved my problem.

Changed followingly:

/etc/haproxy/haproxy.cfg:

global
    -- clip --
    log         127.0.0.1 local2
    -- clip --

/etc/rsyslog.d/haproxy.conf:

local2.=info     /var/log/haproxy/haproxy-access.log
local2.notice    /var/log/haproxy/haproxy-info.log

note also the filename! Another problem was that my ansible script (that was putting these files in place) had a fault where the rsyslog config file missed the '.conf' part!

And then restart the relevant services:

# systemctl restart rsyslog
# systemctl restart haproxy

As a sidenote, tcpdump also lets me see that there is, indeed, syslog traffic:

# tcpdump  -i lo
-- clip --
17:11:35.178564 IP localhost.58466 > localhost.syslog: SYSLOG local2.info, length: 157
-- clip -- 

(when doing a curl 127.0.0.1 from another terminal -- however, such debugging is somewhat unfeasible if there is lots of other traffic going on. I had the luxury of being the only thing/person doing requests against this machine right now.)

Now another thing is that logging is duplicated in /var/log/messages and /var/log/haproxy/. Thanks to a coworker, I got this solved by changing another line in /etc/rsyslog.conf:

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none               /var/log/messages

Changed into:

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local2.none    /var/log/messages
  • If you could please point out the section of that link (into your question) that solved the problem, that would then qualify as an Answer. As-is, this is a link-only Answer and could be deleted. Thank you! – Jeff Schaller Dec 18 '19 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.