0

I have a file that stores some command that looks like this:

sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar

The above command would only run if there is a file name (e.g. file.py) attached to it. So, if I were to run this in the command line, I would input

sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar file.py

However, I have a few files I'd like to run with this command, and I'd like to just specify the file name each time I run it. I tried to save this command in a file named command, and did

cat command echo file.py | bash

but it doesn't seem to work. How should I proceed from here?

1 Answer 1

1

In your script, simply make it a variable with a quick sanity check:

pyfile="file.py"
sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar "${pyfile?python script not specified}"

The construct ${var?message} will throw an error and display message if the variable var is undefined or empty.

You can alternately supply a default value using ${var-defaultvalue}.

You can also make this a function for simpler invocation:

runjob() {
    sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar "${1?python script not specified}"
}

runjob "/path/to/file.py"
runjob "/path/to/some/other/file.py"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .