Currently I work on my personal computer having an i7-3820 processor with 4 cores, 8 hyperthreads.

I run a standalone apache-spark sat with a parameter of local[6] (6 vCPU) to do some calculations. But this have to change, because I would like to have, at the end, a dispatching of components that would be those :

  1. An apache-spark doing calculations. One master and two workers to have a true cluster.

  2. A PostGis (Postgresql) database.

  3. A storage of pain text files or csv files, usually coming from Open Data.

  4. A Geoserver instance to create geographicals maps.

  5. An analyser of logs, ELK.

  6. A Kafka stream to carry events from/to components about what they are doing, discovering, have to do, and analyze logs.

  7. A web application for the users to see maps and results in an Angular application.

    • If I keep a standalone apache-spark like today, and abandon the idea of clusterizing it, I could put them all on my own computer, provider I play with ports and environment variables.

    • I can create many Debian 10 VM on it, using Vagrant and Virtualbox.

    • I can also dockerize each component.

How should I choose between :

  • Many VMs, with Vagrant / Virtualbox.
  • Docker only, a Docker-Engine directly installed on my computer,
  • Some VMs with Docker on it ?
  • Anything better, to consider ?

But aren't I doomed to fail, as :

  • 3 vCPU for an apache-spark cluster,
  • 1 for PostGis,
  • 1 for Geoserver,
  • 1 for ELK,
  • 1 for the web application,

= 7 vCPU on 8 available, leaving me with only one vCPU to run my Eclipse to develop and to do others tasks on my computer ?

  • Neither VMs nor Docker is restricted to assigning specific cores to specific VMs/containers. All VMs or containers can run on all cores (unless you configure restrictions). And using a VM with Docker on it is sort of overkill... in particular if you want to use it as a development environment. – dirkt Nov 7 '19 at 14:23

This type of decision is often much more of a sys-admin decision rather than a developer one. The decision to divide up into VMs / containers is often not one you should be thinking about on a development machine. A lot of the time you are fine to develop "on bare metal", with everything installed directly.


There's a couple of considerations that a sys-admin would prefer you to have thought about as well as things that will help you in the long run.

  1. Since the production version of your software will be deployed onto several nodes (broken up as you have suggested in your question), its good practice to test that everything works in this configuration.
  2. It makes sense for a developer to decide where the dividing lines are between containers go, rather than the sys-admin. For example if it's going to be deployed with docker, it makes sense for the developer to write the Dockerfile whether or not they run it this way on their development machine.

With that in mind, you really don't need to worry too much about the exact nature of your containers, only where the dividing lines between them are.

VM vs Docker vs Docker on VM

As far as I know VM technology is capable of assigning more virtual cores cumulatively across all VMs than the host has actual cores. Though no VM can have more cores than the host has physically - your processor with HT counts as 8 cores not 4. So you might be able to run in VMs if you wanted to. But in general VMs are heavy weight containers. They run full operating system (including Kernel) inside each container and tend to hog a lot of RAM if not CPU.

VMs are also very stateful. One challenge in commercial development is keeping your production environment, test and development environments in line. It's easy to leave a lot of junk in a VM that makes tests pass which will then fail in production.

On the other hand Docker (not running on a Windows host) isolates processes in a way that is very light weight, it does not run a separate Kernel for each container like VMs do. Instead it uses special isolation technology built into the Linux Kernel to run processes on the host. This makes it extremely efficient.

Also Docker emphasizes disposable containers. This makes it very easy to tear-down (destroy) a container and start again from your image to prove that it will work when it gets to production.

Not the answer you're looking for? Browse other questions tagged or ask your own question.