There is no good or reliable way for a host to tell whether another machine is already using its IP address, or if the IP is in the wrong subnet for your local network.
You could use
arpwatch or similar to listen to the network interface to try to find out if some other MAC address is using your IP (but even this is not reliable because the other host may be switched off or have no reason to send any packets on the network at the moment. Also your host is likely to be on a switch - hardly anyone uses hubs these days - so will only be able to see broadcast packets from the other host and not, e.g., communication between the other host and a server).
Even trying to detect packets from different subnets is problematic - it is quite normal and legitimate for the same physical ethernet segment to have multiple subnets transported on it. You may see packets from the alternate subnet but none from your own subnet and assume that your subnet is wrong, when all that has really happened is that there has been no traffic from other hosts on your subnet in the time you were listening.
How long do you want to listen to the network interface for? How long do you consider it reasonable to delay booting and starting network-dependent services while you listen? 5 seconds? not long enough. 5 minutes? that's probably long enough for a decent sample of the network (no guarantees, though) but way too long to wait for your rpi to boot.
It makes more sense to try for a DHCP IP and then fallback to a static address if there is no DHCP server (but if you knew what a safe static IP address to fallback to was, it'd be simpler just to use that).
Stick to using either DHCP or a static address.
If you're the network admin, make sure that there's a working DHCP server on the network. Optionally configure it to give a fixed IP addresses to your rpi's MAC addresses.