Reputation: 619
My code below converts spark dataframe to Pandas to write it as a CSV file in my local.
myschema.toPandas().to_csv("final_op.txt",header=False,sep='|',index=False,mode='a',doublequote=False,excapechar='"',quoting=None)
Output of above command:
"COLUMN DEFINITION|id"|int
"COLUMN DEFINITION|name"|string
Note that in my 'myschema' dataframe there are no double quotes. While writing to CSV double quotes are coming. Desired output is without double quotes as below:
COLUMN DEFINITION|id|int
COLUMN DEFINITION|name|string
I thought setting doublequote=False,excapechar='"',quoting=None
these will solve it. But no luck.
Upvotes: 2
Views: 6614
Reputation: 34046
Pass quoting=csv.QUOTE_NONE
to to_csv
command:
myschema.toPandas().to_csv("final_op.txt",header=False,sep='|',index=False,mode='a',doublequote=False,excapechar='"',quoting=csv.QUOTE_NONE)
Upvotes: 2