If your host doesn't respond to anything, then your options to communicate with it are limited. You aren't going to be able to do things like send it a user name and password, since that would require opening a TCP connection, and that requires traffic in both directions.
You can do port knocking to make the server start responding to a usable protocol such as SSH. Port knocking works by encoding a password into a series of packets that don't require a connected protocol, for example sending pings on a series of ports, or sending an ICMP or UDP packet to a specific port containing a password.
Note that any method that requires the server to be stealthy has an inherent weakness: somebody observing the traffic can see the traffic that leads the server to reply, and replay that sequence. You can limit this by making the password depend on the time, but beware that this can lock you out if the clocks on the client and the server get out of synch.
Obviously the server won't be stealthy while you're communicating with it.
Setting up port knocking isn't that difficult, you can find tutorials for it on the web. But even so it's usually not worth the trouble. Port knocking does not improve privacy and does not defend against any serious threat. Its main advantage is to make your server invisible to generic probes that attempt to exploit security vulnerabilities, but those probes are harmless if you keep your server up-to-date with security patches.