2

We have a spark cluster that launches via supervisor. Excerpts:

/etc/supervisor/conf.d/spark_master.conf:
command=./sbin/start-master.sh
directory=/opt/spark-1.4.1

/etc/supervisor/conf.d/spark_worker.conf:
command=./sbin/start-slave.sh spark://spark-master:7077
directory=/opt/spark-1.4.1

The challenge for supervisor is these scripts launch a daemon process and detach, where supervisor expects things to run in foreground without a fork. So far, my efforts to convince supervisor that forking is okay or to convince spark not to fork have come to naught. Anyone find a better way? Thanks!

4

Solution I inferred from a previous version of the documentation:

/etc/supervisor/conf.d/spark_master.conf:
command=/opt/spark-1.4.1/bin/spark-class org.apache.spark.deploy.master.Master
directory=/opt/spark-1.4.1

/etc/supervisor/conf.d/spark_worker.conf:
command=/opt/spark-1.4.1/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
directory=/opt/spark-1.4.1

Launching via the bin/spark-class command stays in the foreground, and has the added satisfaction of not perpetuating the "slave" terminology.

2

As of Spark 2.1 (and this commit) it is possible to set the environment variable SPARK_NO_DAEMONIZE and have supervisor invoke the provided start scripts $SPARK_HOME/sbin/start-master.sh and $SPARK_HOME/sbin/start-slave.sh directly.

This is documented in $SPARK_HOME/conf/spark-env.sh.template and $SPARK_HOME/sbin/spark-daemon.sh

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.