I am looking at this example from the apache spark documentation. It seems to pass a variable called local[4] into the --master variable. I have not seen that before. What does it mean? I am using bash on OS X. Is there a "local" array? Is it a maven construct?

# Package a jar containing your application
$ mvn package
[INFO] Building jar: {..}/{..}/target/simple-project-1.0.jar

# Use spark-submit to run your application
$ YOUR_SPARK_HOME/bin/spark-submit \
  --class "SimpleApp" \
  --master local[4] \
Lines with a: 46, Lines with b: 23
  • 2
    If that's a command to be run in a shell, then local[4] is of no significance to the shell. The application may care about it. – muru Feb 10 '15 at 3:31

This is a Spark question, not a bash or Maven issue. For Spark, the master name can be in the form:

local[K]: Run Spark locally with K worker threads (which should be set to the number of cores on your machine).

Check out https://github.com/mesos/spark/wiki/Spark-Programming-Guide#master-names for more information.

| improve this answer | |

Square brackets do not have a special meaning in this context are a globbing feature ("character range") in this context, so with default configuration (nullglob off) it will pass literal string local4 if there is a file local4 in the working directory, and local[4] otherwise (thanks @Celada).

This is not a variable reference (${local[4]} would be an array index substitution).

| improve this answer | |
  • 1
    That is only true if there is not a file in the current directory called local4! – Celada Feb 10 '15 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.