Socket activation in systemd can work in two modes:
Accept=true: systemd keeps the listening socket, accepts every incoming connection, spawns a new process for each connection and passes the established socket to it. This case is trivial (each process exits when it's done).
Accept=false: systemd creates the listening socket and watches it for incoming connection. As soon as one comes in, systemd spawns the service and passes the listening socket to it. The service then accepts the incoming connection and any subsequent ones. Systemd doesn't track what's happening on the socket anymore, so it can't detect inactivity.
In the latter case, I think the only truly clean solution is to modify the application to make it exit when it's idle for some time. If you can't do that, a crude workaround could be to set up cron or a systemd timer to kill the service once an hour. This could be a reasonable approximation if the service is only spawned really infrequently.
Note that the use case is probably pretty rare. A process sitting in poll()/select() waiting for a connection doesn't consume any CPU time, so the only resource that's used in that situation is memory. It's probably both easier and more efficient to just set up some swap and let the kernel decide whether it's worth keeping the process in RAM all the time or not.