CODEWITHSUNDEEP
pythonpandasdataframeapache-spark
marmar
marmar

Reputation: 121

Add ascii character to dataframe in spark

Currently have a dataframe where I want to place an ASCII character in the last row. I have a pandas dataframe with about 17 columns, and created a new dataframe with a new row.

newrow = pd.DataFrame(columns=['A','B','C','D'.....and so forth]) newrow.loc[0] = [chr(26),'','','','','','','','','','','','','','','',''] newrow.head()

I then append this to my original dataframe

df= df.append(newrow, sort=False, ignore_index=True)

However, once I save this dataframe as a spark dataframe and output it as a csv, I cannot see the ascii character that I need.

    spark_df = spark.createDataFrame(First_Final)
save_location= "abfss://[email protected]/llanding/P2P/FinalDF"+First
csv_location = save_location+"temp.folder"
file_location = save_location+'.out'

spark_df.repartition(1).write.csv(path=csv_location, mode="overwrite", header="false", sep='\t')

file = dbutils.fs.ls(csv_location)[-1].path
dbutils.fs.cp(file, file_location)
dbutils.fs.rm(csv_location, recurse=True)

Once I refresh in my landing, the csv contains none of the ascii characters I asked for. How do I go about fixing this? Should I add a new row with just the ascii character and append that way despite having not the same number of columns?

Thanks

Upvotes: 0

Views: 207

Answers (1)

Arsh Kenia
Arsh Kenia

Reputation: 143

You can directly use the inbuilt function of pandas : df.to_csv() There is no need to convert it to a spark dataframe.

Upvotes: 1

Related Questions