Yes—the two main approaches are
setrlimit(2) and (on Linux) control groups.
The main advantages of
setrlimit is that it's simple and portable (specified by POSIX). You
fork, then set limits for
exec the code. The downside is that it's per-process, so the code could still fork to exceed the limit (each process it forks will get the same limit, but it's not shared). You could also set
RLIMIT_NPROC to make
fork fail (but of course, then code that needs to fork can't be tested).
The main advantage of control groups (cgroups) is that they can treat the process and all its children together—they get a combined limit. Downside is non-portability (Linux-only), and quite a bit more complexity. You may also have to talk to systemd, if it's managing control groups on your system. The best intro to cgroups is Neil Brown's Control Group Series on LWN.
Another approach is to run the code in a VM, which is then (at least in the case of KVM) a mostly ordinary process to the host. Much more heavyweight, but provides a lot of isolation.
Also, as a side note, you have a lot more things to worry about than infinite loops if you're running arbitrary untrusted code. You'd definitely not want someone to upload a spam bot, for example. I'd suggest looking at existing implementations, but AFAIK they're not open source.