Yes—the two main approaches are setrlimit(2)
and (on Linux) control groups.
The main advantages of setrlimit
is that it's simple and portable (specified by POSIX). You fork
, then set limits for RLIMIT_AS
and RLIMIT_CPU
, then exec
the code. The downside is that it's per-process, so the code could still fork to exceed the limit (each process it forks will get the same limit, but it's not shared). You could also set RLIMIT_NPROC
to make fork
fail (but of course, then code that needs to fork can't be tested).
The main advantage of control groups (cgroups) is that they can treat the process and all its children together—they get a combined limit. Downside is non-portability (Linux-only), and quite a bit more complexity. You may also have to talk to systemd, if it's managing control groups on your system. The best intro to cgroups is Neil Brown's Control Group Series on LWN.
Another approach is to run the code in a VM, which is then (at least in the case of KVM) a mostly ordinary process to the host. Much more heavyweight, but provides a lot of isolation.
Also, as a side note, you have a lot more things to worry about than infinite loops if you're running arbitrary untrusted code. You'd definitely not want someone to upload a spam bot, for example. I'd suggest looking at existing implementations, but AFAIK they're not open source.