Your pager displays its contents in the alternate screen buffer, which selects “alternate scroll mode” (which you can see because the vertical scroll bar changes — it no longer allows scrolling the window contents), and in that mode, the terminal translates mouse wheel movement to cursor up/down controls. This was introduced in XTerm a long time ago and is widely supported in current terminals.
To see what sequence is used to switch to the alternate screen buffer, run
tput smcup | od -a
You should see
esc [ ? 1 0 4 9 h. This is what the pager sends to the terminal. ↓ is represented by CSI B which the terminal sends to the running application.
If your pager is
less, you can disable this by exporting
LESS=X to the environment (this skips terminal initialisation).