I've been asked to create a high throughput router using PC / server hardware. I'm modestly familiar with pf, iptables, and tc, but I've never had such high churn and so many rules. I hope you can help me avoid stupid solutions that won't work.
My goal is to support at least 10,000 simultaneous clients, with at least 100 clients added and removed per second. Each client will have simple bidirectional routing rules with the following functionality:
- IPv6 only
- Forward packet (replacing source and dest addresses)
- Record bytes received
- Stop forwarding after a set amount of data was received
My concern is that creating and destroying 100 rulesets per second will kill performance due to some hidden mutex.
My candidate solutions are:
- OpenBSD with pf. This is my preferred solution.
- Debian with iptables + tc.
- A custom kernel module. I'd prefer not to do this, but I will if I must.
Which of these solutions will scale best? What tricks should I use to get the best performance I can?