Advantage of a 64-bit kernel: if you ever want to run a 64-bit program, you need a 64-bit kernel.
Downside of a 64-bit kernel: with a 32-bit userland, it can be annoying at times, because many build scripts assume that the kernel and the libraries are the same architecture.
Performance-wise, I don't think you'll notice the difference. The 64-bit kernel should be better at crypto, but this would only matter if you did an awful lot of disk and network encryption and your CPU was the bottleneck (it hardly ever is).
So I'd go with a 32-bit kernel in your case.
Install both; if you ever change your mind, the other kernel will only be a reboot away.
See also My processor is 64-bit - does that mean I need the amd64 image?, What version of Ubuntu to use for Desktop with 8Gb RAM? (but they're about choosing between a completely 32-bit system and a completely 64-bit system).