Does the speed of USB-device matter?
It matters for the boot process itself. It has to load the bootloader, kernel and initramfs from USB, which can be several MB altogether, so it can make a difference of a few seconds.
Does the system contiously use the bootloader to decrypt
No. Usually you can pull the USB device out as soon as the first kernel message appears on the screen. That is, unless you also have keyfiles on the USB device, in which case you can pull it only after it passed the Initramfs / starting of services phase.
tl;dr it does not matter for the running system once it's booted
That said, I'm very happy I switched to a faster USB3 stick for booting, but I also use entire LiveCDs on that stick, where USB speed matters a lot more as the whole Live system is on USB in that case.