14

I'm working on a project that implements distributed simulations: arbitrary code is executed on multiple nodes and the results are later collected and aggregated.

Each node is an instance of an Ubuntu Linux virtual machine and runs a master process which takes care of forwarding the code to be executed to a number of worker processes (1 for each core).

This question is about how to make sure that each worker operates in a sandboxed environment without resorting to the use of a virtual machine instance for each of them. The exact requirements for the workers are:

  • fs: no write permission, read-only permission limited to a single directory (and sub-folders)
  • net: only local communications allowed (IPC, TCP, whatever...)
  • mem: cap on memory usage (no swap memory) kill if over mem limit
  • cpu: only 1 core allowed, kill if over time limit

No other limitations should be imposed: the worker should be able to load dynamic libraries (from the read-only folder), spawn new threads or processes, call system function, ecc ecc but the limits must be inherited by the spawned / loaded entities and should apply in a sum-wise way (for instance we can't have a worker spawn two threads that use 800MB each is the memory limit for such worker is 1GB).

It goes without saying that there should be no way for the worker to raise its rights.

I spent considerable time reviewing the available alternatives (SELinux, AppArmor, cgroups, ulimit, Linux namespaces, LXC, Docker, ...) for the simplest solution that satisfies my requirements but my experience on the field is limited.

Current understanding: LXC and Docker a bit on the heavy side for my use case and are not completely secure 1. AppArmor preferable to SELinux due to easier configuration, use it for fs and net restrictions; cgroups preferable to ulimit (which operates on a single process), used it for mem and cpu restrictions.

Is this the simplest way to achieve my goal? Could I use AppArmor or cgroups exclusively? Is there some obvious security hole in my model? The guideline should be "worker allowed to bring down itself but nothing else".

  • 2
    If limit[ing] resources is your goal, you could do a lot better than an Ubuntu guest (or really any Debian-derivative for that matter). In any case, you probably want user-mode linux and/or (with recent kernels) the User namespace – mikeserv Dec 21 '14 at 18:35
  • 2
    LXC sounds like exactly what you need. Why do you think it's on the heavy side and insecure? (Sure, it's had bugs, but so has anything you might use.) – Gilles Dec 21 '14 at 22:30
  • The linked presentation (admittedly from 2011) and the Security section of the Ubuntu LXC documentation talking about 'namespaces leaks' are not very reassuring. Seems like LXC, based mostly on on namespaces and cgroups, could be the best option right now anyway. I also found Linux-Sandboxing, interesting reading – StephQ Dec 22 '14 at 8:31
  • It might require a little bit of retooling, but have you considered running on BSD jails? – Ryder May 30 '16 at 14:24
  • While LXC might be 'heavy' in that it's like a bunch of VM's, it is really simple to make them. Some of these solutions, while 'lighter' might require a lot of configuration. With LXC, you might not need to configure things like writing, since the one app would have the whole container. – MikeP Jul 28 '16 at 3:11
1

Yes, you can use cgroups and SELinux/AppArmor exclusively to monitor and control the arbitrary code that you will execute.

With cgroups, you can do the following:

  1. Limit CPU core usage to 1 CPU with the cpuset subsystem
  2. Set memory usage limits with the memory subsystem, tracking even the forks. See https://github.com/gsauthof/cgmemtime for an example.
  3. Prevent network access to anything that isn't on lo with net_prio subsystem.

And with SELinux/AppArmor, you can limit the process's read/write access.

Note: I am unfamiliar with AppArmor, but it is a Mandatory Access Control (MAC) system, meaning that guarding writing and reading is it's job.

Using these systems is a matter of the writing the proper configurations. Of course, all this is much easier said than done. So here are a few reference links to get you started:

Good Luck!

1

I would discard SELinux for AppArmor only if I were using Ubuntu. (really quite difficult)

LXC is not secure by itself If you need security you must use them via libvirt (based on SELinux MLS).

Your problem is infinite so do not try to find any solution off the shelf and without infinite time, remember that even kernel.org was pwoned and very recently the FBI declared that someone has been using their systems for years without being detected until now.

I will go with LXC/libvirt for pretty good security or I will try the "new" intel clear containers, that use a very light VM for your container with clear use of DAX/KSM (I haven't tested them but they look very promising indeed).

If you're concerned about kernel exploitation, grsecurity is your solution but you'd have to integrate it with your container solution (headaches for sure).

So not easy task for sure, LXC/libvirt are really neat but maybe clear containers are the way to go.

Docker? I haven't/wouldn't use docker for more than local testing when there was no vagrant box available, they need way more work and way better community.

Of course systemd containers are nice also but I assume you don't like/want them cause you didn't even mention them and they are not a vendor agnostic solution.

If you want something "easier" and more amateur you could check out firejail, I've been using it for some desktop "apps" and it does the job (it's quite easy to create the template for your custom app, use "private" mounts on top of your dirs and restrict the network for local use only, spawned processes inherit for parent and goes on...).

Cheers and have fun without going mad. ;)

0

seccomp-bpf is another option which works well for OpenSSH, vsftpd and Chromium it only has exit(), sigreturn(), read() it does also use write() although it allows filtering of system calls using configurable Berkeley Packet Filter rules. It could also be used in conjunction with cgroups for memory, cpu etc...

https://wiki.mozilla.org/Security/Sandbox/Seccomp

0

You may want to look into grid computing systems. In particular, BOINC (http://boinc.berkeley.edu) checks almost all of your boxes.

I believe it operates on your parameters as such:

fs: can read/write to its own directory, nowhere else

net: can be configured to only allow network access to your BOINC server, but isn't default out of the box IIRC

mem: yes, separate memory limits for idle and non-idle machines

cpu: yes, can even say "don't run if computer isn't idle"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.