I own a root server and I would like to log, if any process uses more than X-amount of Ram.
Does anyone know a good script or an existing program for that?
There are no direct ways for that. But you can script easily a nagios plugin for that (but not only for nagios, also for any system monitoring and alarming software what you are using).
If you don't use any, you can make a cron script for that, too.
For the command line to show processes using the most memory, check this answer.