Your assumption is wrong.
The stored data is not encrypted with your passphrase, but with a key that is stored in one out of several (normally eight) key slots. Your passphrase is only used to encrypt/decrypt this key. The key then encrypts/decrypts the data in the LUKS container. When you change that passphrase, the key is re-encrypted with the new password.¹
This also means that you can have up to eight (or whatever the number of slots is) different passphrases for the same LUKS container, all being valid the same time. This way, you could give different people different passphrases and revoke them one by one, if not needed anymore. All key slots save the same key, but encrypted with different passphrases.
Caveat: An important conclusion is that you have not only to remember your passphrase but you should also make a backup of the LUKS header. Otherwise, if there is a fault in the LUKS header area, you won't get your data back, even if you still know you passphrase. (But you should have a backup of your data anyway.)
See details of the LUKS on-disk format here.
¹ Well, to be correct: If there is still a free key slot, the newly encrypted key is saved there and then the old slot is deleted.