I have an OpenBSD server that has a running webserver (the built-in apache web srv).
How can I harden this setup? I want to only server static html files, no php, no sql.
If you're only serving static files, you don't need modify the config. The defaults are secure. Anyway, the OpenBSD FAQ is your friend, particularly the section about apache+chroot.
In the future, nginx will probably replace apache in base.
The OpenBSD devs have already done this work for you! The default install is already hardened and lacks support for php "out of the box" and includes OpenBSD's audits and security changes. The fact that Apache runs w/in a chroot by default also isolates the daemon to the directory /var/www. The default configs in your httpd.conf file are also set to sane defaults, but you should review them for your particular case.
Like most things in OpenBSD, the defaults are sane, and if you want to shoot yourself in the foot, you have to explicitly do so.