When I try to telnet to a port on a server, and if there is no program listening on that port telnet dies with a "Unable to connect ... " error. I understand that. But, why do we need a firewall if there is no program listening on any ports?
There may not be a service running right now, but how about tomorrow? You have them all turned off, but what about your users? Anyone on a unix/windows/mac system can open a port > 1024 on any machine they have access to. What about malware? What about a virus? They can also open up ports and start serving information to the world, or start listening for connections from the network.
A firewall's main purpose is not to block the ports for services you know are disabled, it is to block the ports on services you might not know about. Think of it as a default deny with only certain holes punched in for services you authorize. Any user or program started by a user can start a server on a system they have access to, a firewall prevents someone else from connecting to that service.
A good admin knows what services need to be exposed, and can enable them. A firewall is mostly to mitigate the risk from unknown servers running on your system or your network, as well as to manage what is allowed into the network from a central place.
It's important to know what is running on your machine/server and only enable what you need, but a firewall provides that extra bit of protection against the things you don't know about.
IF there is no program listening on any port, you don't need a firewall, but you also can't connect to your server because it is 'sealed' from the rest of the world.
On the other hand ... let's say your server does not have any locally running program listening on any port, but it serves as a gateway for other computers behind it. In this case you use a firewall to manage the masquerading (NAT) and optionally you can filter some stuff on packet forwarding.
Strictly speaking it may not be necessary, however, keep in mind that a firewall can provide more functionality than simply refusing connections over network ports. For example, DROP versus REJECT behavior.
But, why do we need a firewall if there is no program listening on any ports?
If you have a Single-User Desktop, not a server, you don't need a firewall, if there is no service running, as on a default Ubuntu installation.
Windows had some times, after being able to do networking, some services running on default for maintenance, updates, internal message passing and so on. You couldn't stop them, without stopping windows working, but they where vulnerable for external attacks. So the windows users needed a firewall, and the meme, that everybody needs a firewall, spread fast.
When they met the Linux folks, which often were server admins, they didn't say 'you don't need a firewall on linux' but 'we have free firewalls like iptables for nearly a decade'.
A personal firewall, sitting on the system it shall protect, isn't the best idea either.