Have you tried:
gpg --export-secret-key
That should do it. You might have to give the id of the key you want to export. And maybe also provide --armor if you want it in nice ASCII.
You don't need the public key because the public key can always be trivially generated from the private key.
Keeping a backup is likely a wise idea. I also tend to generate a revocation signature (the --gen-revoke option) that I don't publish and keep that backed up too, often in a separate place. This allows me to revoke the key if I lose it or it becomes compromised in some other way.
I strongly recommend you you keep the key backup offline.