For TCP (though the same approach would work for SCTP1 or any connection-oriented transport protocol), same as for looking for listening ones:
lsof -nPi tcp:the-port
Will report the processes that have a TCP socket open on that port. If you know the source port (your server application can know about it and log it), you can use that instead to pinpoint the rogue client.
For UDP or RAW sockets, it would be trickier though I suppose it's where something like systemtap or dtrace can come handy. Possibly auditd as well.
1 Though SCTP support (on Linux only) was added to
lsof in version 4.86, you cannot use
-i to ask for SCTP sockets explicitly. Here can use
lsof -nP | grep -w 'SCTP.*:the-port' as a heuristic instead.