From the emacs manual
Unlike Shell mode, Term mode does not track the current directory by examining your input. But some shells can tell Term what the current directory is. This is done automatically by bash version 1.15 and later.
It seems this is not done by zsh, at least not by default. Is there an option that I can enable or something?