Ego
Reputation: 585
Option for specifying Spark environment API when using Spark Shell
Is there an option you can pass to the spark-shell that specifies what environment you will be running your code against? In other words, if I am using Spark 1.3; can I specify that I wish to use the Spark 1.2 API ?
For example:
pyspark --api 1.2
Upvotes: 1
Views: 79
Answers (1)
Ashrith
Reputation: 6855
spark-shell initializes org.apache.spark.repl.Main to start REPL, which does not parse any command line arguments. Hence no it will not be possible to pass api value from command line, you have use respective spark-shell binary from their respective versions of spark.
Upvotes: 2
Related Questions
- How to set a custom environment variable in EMR to be available for a spark Application
- How to pass environment variables to spark driver in cluster mode with spark-submit
- Can spark-defaults.conf resolve environment variables?
- Unable to run spark-shell from bin
- Setting Environment variables in Spark Cluster Mode
- Set default packages variable for spark-shell
- In Apache Spark. How to set worker/executor's environment variables?
- Setting Spark configuration through environment variable, command line arguments or code?
- Unable to set Environment Variables in Spark Application
- Spark-shell --conf option for Kryos/Java serializer