If you detect multiple connections using the same local IP+local port combination, then this system was the passive endpoint when those connections formed. When a local port+local IP combo is being used for an outgoing connection, no other process can normally use the same local port with the same local IP for anything else.
Once established, a TCP connection should be fully symmetric, so there may be no attribute associated with just the connection itself that could tell if the connection was established in the inbound or outbound direction: if you detect a listening socket or other connections using the same local IP+port pair you can be pretty sure all those connections are incoming ones.
But if you see just a single connection and no matching listening socket, you cannot be sure. Perhaps it's an outgoing connection, or perhaps it's the last remaining incoming connection to a service that is currently in the process of shutting down and has already stopped listening for new connections.
Some heuristics based on the port numbers can be applied: if the local port number is in the range specified by
/proc/sys/net/ipv4/ip_local_port_range, there is a high probability it's an outgoing connection; or if it's less than 1024, it's probably an incoming one, unless you're using a legacy service that requires an outgoing connection to it must originate from a root-only port (I think some NIS services or something similar has/had this requirement).