There are a few things you can do. My first recomendation would be to visit the Page for open scap and scan your system using the most recent
security guideline configurations available for it. Go through your system and try to get it to at least 90% compliance. Focusing on such things as
openscap and its related security guiders are more focused on a government/DoD environment, they are generally good security guidelines. Having those done as a baseline, will mean that any of the software you install, may run into problems related to the hardening, that will give you a better idea of the challenges involved in setting up software in a secure production environment.
For something like
apache I would recommend just trying to setup a simple
wordpress site, that uses
https if possible. That will also give you some experience in setting up a
sql database. Again once those are done, Try to find some security guidelines to harden them, see what breaks, and learn how to fix it. Learn what allowances have to be made(In a production environment the only 100% secure system is a system that doesn't do anything, so different configurations require different security allowances. That is one of the reasons in a production environment it is best to not have all of your services on one machine. Dividing services among many servers means that not only does your infrastructure not go down in one attack, but there are less openings in any one server to allow an attack)
as far as your direct questions go:
In my production environment I will use
yum install on anything and everything that is available for me to use it. That ensures that my patches are all managed/tested by
redhat/centos/Oracle depending on the Distro being used. So there is a higher likely hood that things will NOT break when patched.
Setup a client machine that is on the same network as your server, and see if it can be accessed through there. In a more true test you'd configure it to be accessible by your local machine too, but that will open you to having to do more work than strictly necessary.
salt. Setup 2-3 VM's that will be
clients/minions, and go about writing
whatever puppet calls their things, that will enforce the security settings I recommended you apply above. That will give you good experience setting up systems as they would be configured naturally, as well as making sure that after security is applied, your salt/puppet/chef servers still can communicate with the clients.
Also, a lot of what you want to learn, is actually covered in pretty good lab scenario's for many study guides for the
Redhat Certified System admin and
redhat certified Engineer tests, looking up labs for that might be a good place to find information too.