Changing the working directory is slightly more involved than just changing a string in the process memory: it's actually kept in the kernel and changed by a system call. So the kernel gets to do an access control check when the working directory is changed. Also, if the
x bit is not set on a directory, you cannot access any files inside the directory, regardless of what your working directory is.
The concept of "executing" a directory doesn't really exist in the way that the concept of executing a program, so calling it the "execute" bit on a directory is a bit misleading. But then, it's often called by other names, the POSIX specification calls it the "execute/search" permission, and I've seen it called the "access" permission. The GNU man pages also call it "search", e.g.
chdir(2) on errors:
EACCES Search permission is denied for one of the components of path.
In the same way you can't "execute" a directory, you also can't "search" or "access" a regular file, so it's only understandable that the same bit was reused in the old times.