What is the current state of GPU accelerated transcoding under linux? I'd like to set up a video converter server and like to use all the available resources besides the CPU. Is there any tool, which can utilize a video card for such a task?
There's nothing currently available under linux that leverages encoding on the GPU, but there's things like vdpau (mencoder/mplayer supports it) on nVidia GPUs that that accelerate the decoding part of the transcoding process, though the large majority of CPU is going to be spent on the encoding side.
There are video capture cards that can encode in codecs like h.264 but it's usually real-time thus the compression and/or quality isn't going to be as good and a fixed set of framerates/resolutions. Devices and drivers for some of these cards can be found at the LinuxTV Wiki.