I'm going to use the words "fast memory" and "slow memory" in this answer. They're very bad descriptions of what really happens, but I think it helps explain what's going on.
Your machine has a "virtual memory" that is made up of fast and slow memory. Your total memory space is the fast+slow memory added together. Now the computer can only work directly on fast memory so the OS uses this first. When fast memory is full the OS will try and move parts of it to slow memory. This frees up some fast memory for use. Now if the program needs to use some data that is in slow memory the OS will move this piece of data from slow memory back into fast memory (it may need to move some data from fast memory out to slow memory first to make space).
This process is "paging" because the system is clever enough that it only needs to move pages of memory and not whole programs. One program can live in fast and slow memory at the same time. The only difference, from your perspective, is that slow memory is slow.
Now when the program that used up all the fast memory has finished there will be lots of fast memory free and stuff will still be in slow memory. But is this a problem? You have the same amount of free virtual memory, and the system will move those pages back into fast memory if they are needed. If no one wants those pages then you might as well leave them in slow memory; it does no harm!
Obviously, in this story, "fast memory" is RAM, "slow memory" is swap.
OK, so that's a simplistic story in "silly" talk.
But it's close to what happens.
Linux doesn't swap programs; swap implies the whole program and data segments are to the swap device in one go. Instead Linux pages; it sends pages of memory to the swap device. This means that a program may have some pages in RAM and some in swap. Pages in swap will be "paged in" on demand if they are accessed; pages in RAM will be "paged out" to swap to create free space in RAM.
So if you ran a program that used up 16Gb RAM and 2Gb swap and then the programs finish you may end up pages left in swap. But only pages that are still allocated to programs will be there. If
free shows 1G of swap in use then you are still using that much virtual memory somewhere. But the pages in swap have not been accessed recently and so may be left there.
Now depending on your work loads you may get better performance leaving those pages on the swap device or you may get better performance pro-actively bringing them back into RAM. But in both cases you have exactly the same amount of free virtual memory.
(Things get more complicated due to Linux allowing overcommit and stuff, but this is the basic idea).
There's a lot of tuning parameters around virtual memory management ( https://www.kernel.org/doc/Documentation/sysctl/vm.txt ) but unless you really need to tune things you might just want to leave the defaults. https://en.wikipedia.org/wiki/Swappiness may be the only one worth looking at.