Linux is quite aggressive in swapping out pages to disk.
This doesn't mean that there is no copy of the page in RAM any more, but it can still be duplicated in the swap cache (a RAM section which caches pages written to the disk).
The advantage of this is, that pages in RAM can be freed immediately in case some process needs memory.
You can check the amount by looking at:
grep SwapCached /proc/meminfo
Also if pages of a process are swapped out, it doesn't mean that those pages are ever needed by the process, but can be from some linked library, which functions/data segments are never used.
Have a look at https://serverfault.com/questions/550793/how-to-find-what-is-using-linux-swap-or-what-is-in-the-swap for a introduction to the topic, there is also a link to scripts to see what is swapped out.
Virtual memory management can be confusing at times, as a rule of thumb:
Usually you shouldn't care about how big the swap is, but how much gets swapped in and out on normal activity (check vmstat).