There are a couple of approaches that you can pursue.
./configure with the same options you originally passed to it and then run
make uninstall. If you no longer have the original files, re-download the tarball of the corresponding version and use that.
locate to track down the relevant files (only really practicable for smaller programmes) and remove them manually.
To avoid this scenario again, you could always look at a tool like checkinstall for building from source and tracking the installed files; it was developed for precisely this reason:
A lot of people has asked me how can they remove from their boxes a program they compiled and installed from source. Some times -very few- the program's author adds an uninstall rule to their Makefile, but that's not usually the case. This is my primary reason to write CheckInstall. After you ./configure; make your program, CheckInstall will run make install (or whatever you tell it to run) and keep track of every file modified by this installation.