So, I want to cycle through previews of windows, not applications with Alt-Tab.
I tried setting Alt+Tab to switch windows in settings, but this made it seemingly do nothing. I did notice that if I were in Firefox with this setting on, when I held Alt+Tab and used the scroll-wheel, it cycled through tab history.