I have implemented SSH rsa key pair for authentication to my linux server (root access is also disabled). Apart from that I have configured iptables to access particular ports by specific ip addresses only. What more I can do to harden the security to my linux servers?
That's a good start, but the list really is endless and depends on what your server will be used for, how it will be accessed, and when, amongst other things.
I recommend using audit software that you install on your server. It will scan your server and give you a score out of 100 as well as provide information on how to harden your server to be more secure. The tool is called lynis, it can be found here:
If you want to go a bit deeper you can take a look at this linux security expert website:
Not all servers are created equal in terms of their security requirements, so as stated above decide what functions and services your server needs to provide and remove any functions or services from the server that won't be used. Once you have done that then install the auditing tool and see the results of running an audit on your server. Study the results and see if any of the recommendations would suit you setup and implement them accordingly.
Keep your server patched by installing security updates often. Limit access to the server to a select few. Make sure you have audit trails in place, the list of hardening possibilities is vast - have fun!