I have a similar question linked over to ServerFault, but after further working on the problem, I believe I've refined the problem to a Linux route table question.
Background:
AWS released their Gateway Load Balancer, which encapsulates packets with GENEVE and forwards them to specified target groups. My goal is to get EC2 instances running Squid to act as transparent proxies for domain filtering. I've got the AWS routing and GLB up and running, and the packets are being delivered to the EC2 instance, but I can't get the packet forwarding behavior to work correctly.
Efforts:
Because the EC2 instance, post-de-encapsulation, should look at it's routing table (once ipv4 forwarding is enabled) and forward the traffic out eth0, as that's the default route.
But, because I need to intercept this traffic, I want to set a next hop for traffic sourced from 10.0.0.0/8 to go to 172.17.0.2 (the IP of the docker container). The docker container has tcp/80, tcp/443, and tcp/3128 exposed, and maps 3128->3128, 80->3129, and 443->3130. Assuming it passes, the traffic egresses from the container, and hits the default gateway of the network (172.17.0.1/16), which then should follow it's local route table to go to the EC2 instance's gateway (10.160.221.97).
It's a bit...complicated, but, the idea is to have the ec2 instance be self-sufficient in receiving the traffic, and filtering it. I don't need stateful tracking, we're just looking to block the outbound traffic for the bad places (no porn in the permit URL list!)
So what I've done so far:
Created the GENEVE interface with a remote unicast address of the GLBe:
#enable geneve tunnel
sudo ip link add name geneve0 type geneve id 1000 remote 10.160.221.119
sudo ip link set geneve0 up
Enabled IP Forwarding:
#set forwarding
sudo sysctl -w net.ipv4.ip_forward=1
Created the custom route table:
#create custom route table with ID 200
sudo echo "200 custom" >> /etc/iproute2/rt_tables
So it should look like this:
cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
200 custom
Added the rule in the custom route table to say if traffic is from 10/8 to use the custom route table:
sudo ip rule add from 10.0.0.0/8 lookup custom
And to make sure my laptop (on the 172.16 network) doesn't lose access:
sudo ip route add 172.16.0.0/12 dev eth0 via 10.160.221.97
(And, y'know, because I'm making this post, you can tell it clearly doesn't work)
So, my route table at default looks like this:
ip route list
default via 10.160.221.97 dev eth0
10.160.221.96/27 dev eth0 proto kernel scope link src 10.160.221.105
169.254.169.254 dev eth0
172.16.0.0/12 via 10.160.221.97 dev eth0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
The traffic comes in to the interface, GENEVE encapsulation, all great. Where I'm getting stuck is the routing on the ec2 instance.
When I add the default route for the custom table, I lose access:
sudo ip route add default via 172.17.0.2 dev docker0 table custom
So, any help or insight into what I'm missing?
Edit - Further Updates:
If I change the commands, no difference:
[ec2-user@proxy-10-160-221-105 ~]$ sudo ip route add default via 172.17.0.2 dev eth0 table custom
Error: Nexthop has invalid gateway.
[ec2-user@proxy-10-160-221-105 ~]$ sudo ip route add default via 172.17.0.2 table custom
