When people discuss the X window system, they mostly focus on the display side, i.e. how it is a networked system that handles and coordinates the graphical output of different applications in the form of windows. Recently, I've had to configure my keyboard layout using XKB (X keyboard extension) and I've learnt that the X window system is also responsible for handling input from the keyboard (and perhaps the mouse too). In my mind, I can imagine the window display and the keyboard/mouse input as being quite orthogonal from each other. But I imagine I must be wrong.
From a design or historical perspective, why does the X window system handle both graphical output and also user input from the keyboard/mouse?