1

I have a program, which cannot be used in multiprocess\multithread mode, so it eat one CPU core, others cores are free, but RAM and network bandwidth usage near to 10% from max.

The server for the given purpose has 8 core CPU, 16GB of RAM, a /20 subnet to serve requests for 4094 external ip addresses.

The main idea was: just run 8 instances, each having a separate tcp port, then just create a redirection rule using some firewall software.

On linux (ubuntu 14.04 LTS amd64) it was done using iptables:

/etc/iptables/rules.v4 (only redirection part)

-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 8 --packet 0 -j DNAT --to-destination :4001
-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 7 --packet 0 -j DNAT --to-destination :4002
-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 6 --packet 0 -j DNAT --to-destination :4003
-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 5 --packet 0 -j DNAT --to-destination :4004
-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 4 --packet 0 -j DNAT --to-destination :4005
-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 3 --packet 0 -j DNAT --to-destination :4006
-A PREROUTING -p tcp -m tcp --dport 4000 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination :4007
-A PREROUTING -p tcp -m tcp --dport 4000 -j DNAT --to-destination :4008

But the program become unstable, and then I tried to compile it for freebsd (releng 11 amd64), and use pf to redirect 4000 port:

/etc/pf.conf

table <mainips> { a.a.a.a/32 }

ext_if="lagg0"

set state-policy if-bound
set optimization aggressive
set skip on { lo0 }
set limit states 150000

scrub in inet all fragment reassemble random-id
scrub inet all reassemble tcp

scrub in inet6 all

rdr pass log on $ext_if inet proto tcp from any to x.x.240.0/20 port 4000 -> x.x.240.0/20 port 4001:4008 round-robin

pass in on $ext_if inet proto tcp to x.x.240.0/20 port 4000

pass on $ext_if inet proto icmp
pass on $ext_if inet6 proto icmp6
pass in on $ext_if inet proto tcp to <mainips> port { 22 }
pass out on { $ext_if } inet proto tcp all flags S/SA modulate state
pass out on { $ext_if } inet proto udp all

#block all

But load balance cycle only thought ip addresses, not ports, so i tried to use ipfilter\ipnat:

/etc/ipf.rules:

pass in quick on lo0 all
pass out quick on lo0 all
pass out quick on lagg0 proto tcp from any to any port = 22 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4000 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4001 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4002 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4003 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4004 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4005 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4006 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4007 flags S keep state
pass out quick on lagg0 proto tcp from any to any port = 4008 flags S keep state

/etc/ipnat.rules:

rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4001 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4002 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4003 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4004 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4005 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4006 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4007 tcp round-robin
rdr lagg0 x.x.240.1/20 port 4000 -> x.x.240.1-x.x.255.254 port 4008 tcp round-robin

This configuration allow to get cycle thought destination ports, but redirect all they only to x.x.240.1 address.

The main question: how to get on freebsd redirection behavior the same as on linux?

migrated from serverfault.com Dec 3 '16 at 9:12

This question came from our site for system and network administrators.

  • Have you considered running 8 processes with the same TCP port and avoid requiring a load balancer? – Richard Smith Dec 3 '16 at 9:57
  • No, because it shows 'Address already in use'. – user1024 Dec 3 '16 at 10:17
  • 1
    You need to set SO_REUSEPORT in setsockopt to allow processes to share the same port. – Richard Smith Dec 3 '16 at 11:18
  • Also, have you considered using a real load balancer like haproxy instead of iptables hacks? – Celada Dec 3 '16 at 13:25
  • you can run daemons on same port, but on different local addresses e. g. 127.0.0.10 ... 127.0.0.17 and than use pf rule rdr pass on $ext_if proto tcp from any to $ext_ip port 4000 -> <app> where app is table with addresses to which your daemon is binded. – citrin Dec 3 '16 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.