Another way you can limit access to sshd on a GNU/Linux system at the socket level with a built-in (assuming init is systemd 235+ and kernel 4.11+) is by utilizing systemd with cgroup/eBPF access lists
Modify the base sshd systemd stanza
sudo systemctl edit sshd
Append the sshd [Service] stanza to your liking
#requires systemd 235+ and kernel 4.11+...