You can use a firewall for that. Disable all incoming and outgoing traffic by default. Allow outgoing traffic on the openvpn network interface. Allow the access to your vpn server from all interfaces (so you can connect to vpn).
If you can only use a host name to access the vpn server, you'll have to make these rules more permissive. Allow outgoing port 53 for name resolution and an outgoing vpn port.
I will put the configuration commands for ufw
(uncomplicated firewall). These commands are based on askubuntu "UFW for OpenVPN" answer.
# Adapt this value to your config!
VPN_ADDRESS=...
ufw --force reset
ufw default deny incoming
ufw default deny outgoing
ufw allow out on tun0
ufw allow out to $VPN_ADDRESS
ufw enable