CODEWITHSUNDEEP
apache-sparkloggingcommand-linepyspark
LC-datascientist
LC-datascientist

Reputation: 2096

How do I log the spark-submit command line that I typed?

How do I write a PySpark script that will log the spark-submit command line into its log output?

For example, when I run: spark-submit script.py arg1 arg2 --flag arg3 --out output

in addition to running its task, I want this command to record that command line into a log file called output.log so I can easily keep track of how I ran it.

Upvotes: 0

Views: 1619

Answers (2)

wbmrcb
wbmrcb

Reputation: 378

Try creating a shell script to execute spark-submit, with given arguments

@echo off
echo spark-submit script.py %1 %2 --flag %3 --out output >> output.log 
spark-submit script.py %1 %2 --flag %3 --out output >> output.log

Upvotes: 1

Ayush Vatsyayan
Ayush Vatsyayan

Reputation: 2616

Create a bash script or python script to execute the command. This will act as a wrapper within which you can log the output to a file. In python you can use os.system() to execute spark-submit

Upvotes: 0

Related Questions