RestartSec seems to only be used if that particular service is the one being restarted via the systemctl restart command.
For example, I have two services, A and B.
If you do a systemctl restart B, it works as expected... but if you do a systemctl restart A, both services are stopped and immediately started again, with no delay.
Systemd is apparently only using the configuration values for the one service you specify, and ignoring them for any dependencies.
This is not as uncommon as it sounds. If B talks to a remote server, starting and stopping quickly may fail due to the remote end rejecting the client. But restarting A directly will happen whenever A is updated without a change to B.
You can probably work around this by adding the delay to A as well, but you shouldn't NEED to do this, as it breaks object isolation by making A know about B when it's not a dependency.