Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Introduction

I have following load balancing configuration:

10.0.1.31 - lb
10.0.1.35 - virtual IP
10.0.1.32 - node1 (tomcat + mysql)
10.0.1.33 - node2 (tomcat + mysql)

I'm using keepalived which is redirecting packets to active nodes - shared ip address is 10.0.1.35. lb has required config:

echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

node1 and node2 to handle properly incoming packets needs to have 10.0.1.35 on lo

[root@lb-node1 ~]# ip addr list dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   inet 127.0.0.1/8 scope host lo
   inet 10.0.1.35/32 scope global lo
   inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever

Problem

Due to the fact that we have 10.0.1.35 on the lo if the locale tomcat is trying to connect to mysql via lb - connection is established to local instance.

I would like skip lo for outgoing packets

Test

From node1 which has disabled mysql service and 10.0.1.35 ip on lo interface I'm trying to connect to mysql on the node2. Unfortunately the result is

   [root@lb-node1 ~]# telnet 10.0.1.35 3306
   Trying 10.0.1.35...
   telnet: connect to address 10.0.1.35: Connection refused 

of course if I remove 10.0.1.35 IP from lo interface I'm able to connect to mysql instance on node2

Solution(?)

I was trying do add routes with appropriate metrics but it doesn't help :/

    [root@lb-node1 ~]# route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    10.0.1.0        0.0.0.0         255.255.255.255 UH    0      0        0 eth0
    10.0.1.0        0.0.0.0         255.255.255.255 UH    100    0        0 lo
    169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
    0.0.0.0         10.0.1.1        0.0.0.0         UG    0      0        0 eth0

@Patrick Solution

vip=10.0.1.35

ip route add local $vip dev lo table 10 proto kernel scope host
ip rule add to $vip lookup 10 prio 1
ip route del local $vip dev lo table local
ip rule add to $vip iif lo lookup main prio 0

@Patrick Solution Issue

Initial state

VIP + your ip route configuration, mysql is working on both nodes


P Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.1.35:http rr persistent 6
TCP  10.0.1.35:mysql rr persistent 6
  -> 10.0.1.32:mysql              Route   10     0          0
  -> 10.0.1.33:mysql              Route   10     0          2
UDP  10.0.1.35:snmptrap rr persistent 6
  -> 10.0.1.32:snmptrap           Route   10     0          0
  -> 10.0.1.33:snmptrap           Route   10     0          1




root@lb-node1 ~]# mysql -h 10.0.1.35 -u test -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 126
Server version: 5.5.36-MariaDB-wsrep-log MariaDB Server, wsrep_25.9.r3961

Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW VARIABLES WHERE Variable_name = 'hostname';
+---------------+----------+
| Variable_name | Value    |
+---------------+----------+
| hostname      | lb-node2 |
+---------------+----------+
1 row in set (0.00 sec)

MariaDB [(none)]> 


[root@lb-node2 ~]# mysql -h 10.0.1.35 -u test -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 133
Server version: 5.5.36-MariaDB-wsrep-log MariaDB Server, wsrep_25.9.r3961

Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

ariaDB [(none)]> ;
ERROR: No query specified

MariaDB [(none)]> SHOW VARIABLES WHERE Variable_name = 'hostname';
+---------------+----------+
| Variable_name | Value    |
+---------------+----------+
| hostname      | lb-node2 |
+---------------+----------+
1 row in set (0.00 sec)

MariaDB [(none)]> 

As you can see everything is working correctly.

Issue

But when I shut down currently active mysql server:

[root@lb-node2 ~]# service mysql stop
Shutting down MySQL.... SUCCESS! 
[root@lb-node2 ~]# 



Every 2.0s: ipvsadm -l                                                                                          Fri May  9 10:20:49 2014

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.1.35:http rr persistent 6
TCP  10.0.1.35:mysql rr persistent 6
  -> 10.0.1.32:mysql              Route   10     0          0
UDP  10.0.1.35:snmptrap rr persistent 6
  -> 10.0.1.32:snmptrap           Route   10     0          0
  -> 10.0.1.33:snmptrap           Route   10     0          1

I cannot connect to mysql from both nodes

[root@lb-node2 ~]# mysql -h 10.0.1.35 -u test -p
Enter password: 

..... 

It seems to me that node1 is not accepting incoming packets, because load balancer is correctly redirecting packets

[root@lb-node1 ~]# tcpdump -i eth0 'port 3306' and src 10.0.1.33 or dst 10.0.1.33
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:35:26.915640 ARP, Reply 10.0.1.35 is-at 52:54:00:30:a3:4b (oui Unknown), length 28
10:35:26.915987 IP 10.0.1.33.38517 > 10.0.1.35.mysql: Flags [S], seq 2024730796, win 14600, options [mss 1460,sackOK,TS val 1298907 ecr 0,nop,wscale 7], length 0
10:35:27.914788 IP 10.0.1.33.38517 > 10.0.1.35.mysql: Flags [S], seq 2024730796, win 14600, options [mss 1460,sackOK,TS val 1299907 ecr 0,nop,wscale 7], length 0
10:35:29.914784 IP 10.0.1.33.38517 > 10.0.1.35.mysql: Flags [S], seq 2024730796, win 14600, options [mss 1460,sackOK,TS val 1301907 ecr 0,nop,wscale 7], length 0

What is strange because still I have VIP on lo

[root@lb-node1 ~]# ip addr list dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 10.0.1.35/32 scope global lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever



[root@lb-node1 ~]# ip route
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.32 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 10.0.1.1 dev eth0 
share|improve this question
    
In regards to your added info, can you remove the 'port 3306' filter from the tcpdump? I want to see if the system is sending any sort of ICMP packets back. If not it means that your system doesn't think anything is listening on that port. Can you do netstat -tpnl|grep mysql? –  Patrick May 11 at 7:30
add comment

1 Answer 1

I dug up some really old code of mine which did something similar to what you're trying to accomplish. Here's what you can do to get the effect you want:

vip=10.0.1.35

ip route add local $vip dev lo table 10 proto kernel scope host
ip rule add to $vip lookup 10 prio 1
ip route del local $vip dev lo table local
ip rule add to $vip iif lo lookup main prio 0

This will cause the box to send any traffic to 10.0.1.35 out to the network (and thus to the load balancer which is answering ARP requests for that IP). But the box will still accept any traffic which the load balancer gives it.


Explanation

I'm going to explain them out of order as they make more sense that way.
 

ip route del local $vip dev lo table local

This removes the route which says to send all traffic to 10.0.1.35 to the local host through lo.
 

ip route add local $vip dev lo table 10 proto kernel scope host

This replaces the one we just deleted, but it puts it in a brand new table (10).
 

ip rule add to $vip iif lo lookup main prio 0

This tells the system that when sending traffic from the local box (iif lo) to 10.0.1.35 to use the main route instead of the 'local' route. This will make 10.0.1.35 traffic pick up the route for the 10.0.1.0/24 subnet (or whatever your local subnet is if not /24).
 

ip rule add to $vip lookup 10 prio 1

This rule is added with a lower priority (higher number) than the above rule so that it matches after. If the traffic didn't match the above rule (not iif lo) it will check table 10 for a matching route, which will pick up the route we added earlier.
The reason for the rule (and the route we added to table 10) is so that any traffic which the load balancer sends to this box (thus not iif lo) won't be rejected. Basically it tells the kernel "yes, this IP belongs to me".
 

The reason we moved the 'local' route into table 10 is because we want our iif lo lookup main prio 0 rule to match. But the 'local' table has highest priority, so it always matches first.

 

The reason we add the rules in the order we do is to prevent interruption. If you added them in the order covered in the explanation, you would have a gap where there is no route to 10.0.1.35 which would result in any 10.0.1.35 traffic being sent to the local box getting rejected.

share|improve this answer
    
Many thanks Patrick! This is exactly what I meant and it works brilliantly :) –  hicolour May 9 at 7:48
    
unfortunately I have another issue closely related with your proposal, your solution is working until the first switch of the VIP occurred. I will try to describe it in original question. –  hicolour May 9 at 9:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.