Mode_switch is the old-style (pre-XKB) name of the key that is called AltGr on many keyboard layouts. It is similar to Shift, in that when you press a key that corresponds to a character, you get a different character if Shift or AltGr is also pressed. Unlike Shift, Mod_switch is not a modifier in the X11 sense because it normally applies to characters, not to function keys, so applications only need to perform a character lookup to obtain the desired effect.
ISO_Level3_Shift is the XKB version of this key. Generally speaking, XKB is a lot more complicated and can do some extra fancy stuff. XKB's mechanism is more general as it allows keyboard layouts to vary in which keys are influenced by which modifiers, it generalizes sticky (CapsLock-style) and simultaneous-press (Shift-style) modifiers and so on.