I would like to take a video of what would run on the X Server on a remote machine and store that on my local machine. I thought of forwarding the X Session and using a screen capture tool, but ideally, I'd like to do this completely from the terminal and output a video.
It seems to me that you're looking for a combination of two things:
Xvfb, a virtual framebuffer X11 server. It behaves just like an ordinary X server, but it needs no real display or even a GPU. It only needs a chunk of memory it can use as a framebuffer.
a version of ffmpeg with the x11grab support compiled in.