For video data
You can use the -dvd-video option to genisoimage/mkisofs to enable the creation of UDF structures required for video playback. Normally this is done in one operation with growisofs (which calls genisoimage itself), e.g.
$ growisofs -dvd-compat -Z /dev/dvd -dvd-video /path/to/dir
however you could use genisoimage by itself to create an image locally, e.g.
$ genisoimage -o test.iso -dvd-video /path/to/dir
Note that in both cases it is up to you to ensure that the directory structure under /path/to/dir is correct for a DVD filesystem, i.e. it includes the VIDEO_TS and AUDIO_TS directories. Using a DVD authoring tool such as dvdauthor will create the required content.
For general data
In the case of writing arbitrary data to a UDF filesystem, the -dvd-video option of genisoimage is not useful. In this case, the mkudffs tool from the udftools package can be used, along with a loopback device to generate a local image.
First, you need to generate a blank file to contain the image. Here the size is given as 650 MB to match a standard CD, this will obviously need to be larger for a DVD/Blu-Ray. See this wikipedia article for the exact sizes of DVD media.
$ truncate -s 650M /tmp/cdimage.udf
Now invoke mkudffs to generate a UDF filesystem in this blank image:
$ mkudffs --media-type=dvd /tmp/cdimage.udf
See mkudffs(1) for other possible media types. Both truncate and mkudffs will create sparse files if your filesystem supports them; so the image won't occupy the full size unless you fill it.
Then you can mount your image locally to copy data to it
$ sudo mkdir /media/udfimage
$ sudo mount -t udf -o loop,rw /tmp/cdimage.udf /media/udfimage
After the data has been copied to the image, the process is followed in reverse to unmount the image and detach the loopback device:
$ sudo umount /dev/loop0