I'm using this on a daily basis. You need an xorg.conf file, in your Device section you need the following:
Option "ExactModeTimingsDVI" "ON"
Then in your Monitor section place your HorizSync, VertRefresh, and ModeLine
Finally in your Screen section add this:
Option "AllowEmptyInitialConfiguration" "True"
Option "ConnectedMonitor" "DFP"
Option "UseDisplayDevice" "DFP-0"
Option "ModeValidation" "AllowNonEdidModes"
Modes "<Resolution and Refresh from your modeline>"
What this does is tells the Nvidia driver that there is a monitor attached, and gives it the information required to start drawing to the frame buffer. No monitor required. I do recommend using x11vnc to do your viewing because it polls the screen and is much more responsive to screen changes than any other linux vncserver.