I think the best way to make use of your cores in GPU is to use OpenCL. The idea is quite simple. You write a kernel (a small block of code, where you can use only basic C code without libraries). For example, if you want to filter a frame, you have to do some calculations on each pixel and that is what the kernel code will do.
Then you have to compile the kernel, allocate the memory on GPU and copy data from the main memory there. You also have to allocate memory for the result. Then you create threads and send the kernel into execution on GPU (I think you can also use both CPU and GPU cores at once to execute kernels). So each thread executes the kernel once for every pixel. After you copy the result back to main memory and continue working with the CPU.
This is the simplest way I can explain this, but there is still a million details you have to know, so you better start learning ;) You can start with your graphics card manufacturer developer web site. There you will find the libraries and tutorials on how to start developing in OpenCL.