It really depends on what you are trying to accomplish, and what you mean by "Distributed Computing Cluster." I did a similar thing once in Uni using old machines and PVM that's the "Cluster" in the sense of a bunch of machines acting as one single computer to do parallel processing - think Beowulf clusters. Of course, you will need code that is written to take advantage of this.
A good place to start, would be determining what you are looking to learn with this project. I recommend reading the wikipedia article on Parallel Computing for starters, and then refining your needs based on what you want to do.
A simple job queuing system (like gearman) may be enough to get some cool results quickly.
The problem I had when I made a parallel computing cluster, was I didn't have anything to do on it, it just basically sat there, but it was a fun project and I learned quite a bit. In any case, you are likely to learn quite a bit and have fun at the same time, regardless of what you choose to implement.
As far as choice of distributions, I would go with what I was most comfortable with, as you will likely need to install things from source. Once you are comfortable getting everything setup, then you can look into finding a distribution that is tailored more towards your needs. But any distribution should do.
What software to configure the cluster? This depends entirely on what type of cluster you create.
On board vs. distributed FS? Again, this depends on what the requirements for your cluster are. Will each node be passing data back and forth among the other nodes? Will they operate as slaves with a single master? will they operatae completely independently? These questions will start to inform your choices. And of course, there are always trade offs.
Some other links that might prove interesting:
http://hadoop.apache.org/
http://www.csm.ornl.gov/oscar/
https://computing.llnl.gov/tutorials/parallel_comp/
http://www.google.com/Top/Computers/Parallel_Computing/Programming/Environments/
http://www.google.com/Top/Computers/Parallel_Computing/Beowulf/
