If I'm moving data on the SAME drive but a different partition, shouldn't it be fast? I assumed the move would be a fat table change...
No, because a FAT is part of a file system, and each partition contains one filesystem. So if you move data to a different filesystem, the operating system cannot simply rearrange things in a fat table -- there are two to consider, and they do not map each other arbitarily. The destination must allocate some of its own space, and the source (in a move) frees some.
If it were just a matter of rearranging the tables, you would run into inconsistencies such as:
I have a 100 GB partition and a 2 GB partition. If moving one to the other just involved rearranging tables, I should be able to move a 20 GB file from the former to the latter.
I move files to a partition on a USB stick, then I move the stick: if moving files just involved rearranging tables, where are the files going to be when I stick this in another computer?
I realize the second case is not part of the context you are referring to, but the reason they amount to the same thing is because otherwise you would require another abstraction layer stored on the device. It cannot be something simply invented and juggled by the operating system, because you may move the device and/or use it under a different OS: now where is the information?
Devices may contain meta data indicating the size, type, and offset of their partitions. Fortunately, they do not contain information about the content of these partitions. I say fortunately because this is bound to create more problems than it solves.
Filesystems are intended to be top level, discrete entities, not things that are part of a larger system of storage (although they may be that in some contexts).
Some devices such as SSDs may implement an optimizing feature akin to what you imply on a hardware level, however. In other words, if you move something from one partition to another on an SSD, it may only rearrange some references, in so far as that hardware is doing accounting for itself as a whole irrespective of how it has been broken into different partitions on a higher level of abstraction. This would be totally opaque to the operating system and everything else, but you may notice it as an extremely fast move. It requires that the device run some kind of firmware which presents a virtual set of block addresses to the operating system, then maps them to the physical itself, which traditional drives do not do: they present the actual physical addresses to the operating system so that it may make whatever optimal use of this that it can. Hence, file system implementations (FAT, etc.) must assume they are organizing an actual physical region of a device and there is no layer above the filesystem to try to further organize the contents of the entire device (beyond breaking it into partitions).