CODEWITHSUNDEEP
pyspark
littlely
littlely

Reputation: 1418

Exception in thread "main" java.lang.IllegalArgumentException: pyspark does not support any application options

When configure jupyter notebook in pyspark, it has something wrong: in the .bashrc file:

export PATH="/root/anaconda3/bin:$PATH"
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --ip=my ip"

when I input pyspark, it shows [C 13:51:52.029 NotebookApp] Running as root is not recommended. Use --allow-root to bypass.

and I use pyspark --allow-root, but it shows :

Exception in thread "main" java.lang.IllegalArgumentException: pyspark does not support any application options.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:290)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:147)
at org.apache.spark.launcher.Main.main(Main.java:86)

what is wrong with it?

Upvotes: 0

Views: 3127

Answers (1)

Juncheng ZHOU
Juncheng ZHOU

Reputation: 125

Replace your .bashrc file as:

export PATH="/root/anaconda3/bin:$PATH"
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --ip=my ip --allow-root"

Upvotes: 1

Related Questions