We have a spark cluster that launches via supervisor. Excerpts:
/etc/supervisor/conf.d/spark_master.conf: command=./sbin/start-master.sh directory=/opt/spark-1.4.1 /etc/supervisor/conf.d/spark_worker.conf: command=./sbin/start-slave.sh spark://spark-master:7077 directory=/opt/spark-1.4.1
The challenge for supervisor is these scripts launch a daemon process and detach, where supervisor expects things to run in foreground without a fork. So far, my efforts to convince supervisor that forking is okay or to convince spark not to fork have come to naught. Anyone find a better way? Thanks!