The vbox video module has to be in the guest. You have to mount the additions iso in the vbox control panel, which will then show the iso cd contents on boot after you mount the cd, then you run the vbox additions installer from the iso contents, which installs all the required drivers. Once the drivers are installed in the guest, vbox will be good to go. I never use the debian packages for my debian vms, I use the iso installer directly.
It's slightly unclear from your question about what is in host and what is in guest. So I'll treat this in two sections, the first assumes no guest additions have been installed, or have been incorrectly installed, and the second is how to actually enable resizable screens. Jump down to the last section first in case your debian guest packages are all fine and there is no issue, look for the enable screen resizing option to be non grayed out, if it is, your guest modules are fine, if it's not, proceed to the first steps and get the modules properly installed.
As an aside, but also possibly the actual answer to your question, my /var/log/Xorg.0.log on this vbox install also says that vboxvideo is failing/unloaded, because module is not installed, but when I check with lsmod, it is, and my screen display resolution once I corrected the vbox driver issues proves it is, so that Xorg.0.log data may be an xorg bug, unrelated to your issue. Xorg.0.log however says that the actual driver is "modesetting" even though it's not loaded, so my suspiction is that there is an issue in Xorg/vboxvideo that makes xorg think the vboxvideo driver is not available when it is. You can see this by simply using lsmod on the booted install. Also you can see if it the advanced screen resize features work or are enabled, if they are, the driver is working.
Installing Guest Additions:
I never use the debian packages in the guest because I prefer to install vbox directly from Oracle, and for the guestadditions to likewise come directly from oracle, to match all the versions, but if you use debian packages, I believe it's really just virtualbox-guest-x11, virtualbox-guest-utils (though your xorg log output suggests that in fact that was failing). Also, you don't have to learn every distro's vbox guest package ins and outs if you use the stuff directly from oracle, you just run the guest additions installer .run package. I've used that method for about 10 years, and never had an issue with it.
I never use the guest distro packages, as noted, so I can't say anything about why they would fail, nor can I debug issues with them (that's why I use the direct install, it's always reliable unless there is a kernel support issue, but that will impact the debian packages too). Basically all the debian packages are is a wrapper around what guestadditions installer would have installed in the first place, that's really all those packages do, which means just another possible point of error or failure.
Take the host vbox master window, select your guest, then select settings, then make sure you the iso is mounted as an optical drive.
Then navigate to the location the iso is at. In Debian Stretch host, with vbox 5.1, that iso is at (use locate or similar tool to find the file):
Mount this iso, boot the guest, then mount the cd image in the guest.
# mount /dev/cdrom /media
ls -w 1 /media
then run it:
This runs the installer, and will install the correct guest additions for the host system and guest. Reboot the guest, if necessary.
The host vbox version should match with the guest additions iso contents, which means if you install vbox directly, along with the guest additions download, in the host, you know that's the right version, and if you then also install the guest drivers from the iso files directly, you know those are right. I'm updating my vbox to check this since I don't use it much anymore.
Installing Required Packages to Build Kernel Modules:
NOTE: on installing guest additions, you have to have: make gcc
plus your kernel headers installed, that's the problem i had. If you see vboxadd install fail message on the guest additions install, that's why.
So install the linux-image, linux-headers, gcc, and make, reboot, and then run (this actually will just run by itself after kernel reinstall):
service vboxadd setup
again, or better, install kernel/ headers/gcc/make before you start, then the vbox guest additions installer will get it all running correctly, and the drivers will get installed, the enable resize switch will be active, and you can set your guest to whatever size you want. Actually, as soon as I installed the latest kernel image/header packages, vboxadd ran automatically, so once I reboot, I had access to all the formerly grayed out controls, full resolution, copy paste etc.
Enabling Resizable Guest Windows Once GuestAdditions Modules are correctly installed
These last two things might be all you needed all along, but the overall answer should help many people who have failed to actually install the guest additions properly in the first place, or who, like me, forgot to install or check that gcc, make, and kernel headers are also installed.
There's a few details I'm not sure about, like setting the desired screen size, since I never use that feature beyond resizing the screen on boot, so here's how to do that.
https://forums.virtualbox.org/viewtopic.php?f=2&t=68966 that vbox forum thread covers specific questions about getting larger resolutions.
First, make sure you are not in fullscreen mode: i.e. switch to your windowed 1280x1024 mode (or whatever windowed mode suits you).
IMPORTANT. Make sure that View|Auto resize guest display is enabled (checked).
Make sure that the Guest Additions are installed, not merely having the GAs CD mounted. If they are installed then there should be an icon on the guest taskbar which will tell you the GAs version and status.
Carefully drag the bottom edge of the VM window to reduce the screen height. You are trying to achieve a 16:9 ratio, so with a 1280 display width you want a display height of 720, exactly.
Shut down the VM completely (not suspend), then restart it. We want windows to start up, seeing a 16:9 ratio while booting up. This should make it happy to accept other 16:9 ratios.
I couldn't initially find the autosize option, it was grayed out, in-active until I actually got the vbox modules built. It's an option in the guest container top menu, view, but mine was grayed out since the module wasn't actually built until I corrected the missing kernel headers/gcc/make issue.
Assuming you installed the modules following the above method, you'd then reboot. After the system is fully started (at login screen, that is), you'll see the option in the guest container window: View -> AutoResize Guest Display, no longer grayed out. Enable it.
Then go down to: View -> Virtual Screen1 -> select the display size you want.
And there it is:
System: Host: vm-openbox Kernel: 3.13.0-107-generic i686 (32 bit gcc: 4.8.4)
Desktop: Openbox 3.5.2 dm: lightdm Distro: Ubuntu Trusty Tahr (development branch)
Machine: Device: virtualbox System: innotek product: VirtualBox v: 1.2 Chassis: Oracle type: 1
Mobo: Oracle model: VirtualBox v: 1.2 BIOS: innotek v: VirtualBox date: 12/01/2006
CPU: Single core AMD Athlon 64 X2 5000+ (-UP-) speed: 2594 MHz (max)
Graphics: Card: InnoTek Systemberatung VirtualBox Graphics Adapter bus-ID: 00:02.0 chip-ID: 80ee:beef
Display Server: X.Org 1.14.5 drivers: (unloaded: fbdev,vesa) Resolution: email@example.com
GLX Renderer: Gallium 0.4 on llvmpipe (LLVM 3.3, 128 bits)
GLX Version: 2.1 Mesa 10.0.0 Direct Rendering: Yes
Network: Card: Intel 82540EM Gigabit Ethernet Controller
driver: e1000 v: 7.3.21-k8-NAPI port: d010 bus-ID: 00:03.0 chip-ID: 8086:100e
Drives: HDD Total Size: 8.6GB (31.4% used)
Info: Processes: 94 Uptime: 3 min Memory: 106.2/908.8MB
Init: Upstart v: 1.11 runlevel: 2 default: 2 Gcc sys: 4.8.4
Client: Shell (bash 4.2.451 running in xfce4-terminal) inxi: 2.3.2