The most immediate downside of running a realtime process is that the process can easily starve out every other process on the system. The result from your point of view will be that the computer is completely unresponsive to keyboard, mouse, and probably network, for as long as the realtime process is using the CPU. This can happen if something goes wrong and the process goes into an infinite loop, or even temporarily if the process starts a long-running calculation without waiting for input periodically. (So, for example, don't run SETI@home with realtime priority.)
A lone single-threaded process on a multi-core CPU is less likely to cause this problem since there are other cores that the lower-priority process can use. But if that process creates any child processes, they will inherit the same realtime priority, so things can get out of control if you aren't careful.
sched_setscheduler(2) man page has good advice:
Since a nonblocking infinite loop in a process scheduled under SCHED_FIFO or SCHED_RR will block all processes with lower priority forever, a software developer should always keep available on the console a shell scheduled under a higher static priority than the tested application. This will allow an emergency kill of tested real-time applications that do not block or terminate as expected. See also the description of the RLIMIT_RTTIME resource limit in getrlimit(2).
That should be a shell on the console -- not under an Xterm, unless you want to give all of X realtime priority as well.