I am building a system which has the functions of an online judge system. I need to run all the executables and evaluate their output. The problem is that if all of them will be placed in a container, in different folders one of the application may try to exit it's folder and access another folder belonging to another application. In this case the main server will be protected, but not the other applications and not the evaluator.
I have found myself a solution, but I am thinking there is a better one, I will create for example 5 containers, each one of them will be runing the same algorithm and each one of them will evaluate 1 problem at a time. After the problem is evaluated this one will be deleted and another one received. In this case, the main server and all the applications will be protected, but not the evaluator. The evaluated application may exit it's folder and start writing random text files for example, filling the entire memory.
The evaluator will start the executable, measure it's time (if it is longer than 1 or 2 seconds it will kill it) and it's used memory(if it reaches a certain limit it will kill it).
I have also thought to create a container each time and delete it after the executable is evaluated, but it takes a few seconds only to create and start the container...
How do I isolate the evaluated process from messing with the container and evaluator? I basically want to block a process from accessing other folders.