The answer can be swap under certain circumstances, but not exactly the ones you're describing (because you're talking about closing the applications). Instead, it's probably the more general phenomenon, paging.
When an application is opened for the first time, its binary needs to be loaded in to memory from disk. All the libraries it depends upon will also need to be loaded, unless another program that uses them has already caused them to be loaded. The second time it's loaded, those binaries and libraries are probably still hanging out in RAM unless the system has been under memory pressure in the interval. So it's not so much a matter of time (maybe more than an hour, as you say) as it is a matter of what else has been using the RAM.
If you leave an application open but unused while something else demands memory, that's when swap can come in to play: the original application's heap pages may be written out to swap while it's unused, and then they have to be paged back in when the application is active again.
How can I help this?
Get more RAM! :)