CODEWITHSUNDEEP
csvdataframeheaderapache-spark-sql
Unix
Unix

Reputation: 89

Pyspark Dropping the header in a dataframe, AttributeError: _jdf

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

spark = sqlContext.sparkSession
avg_calc = spark.read.csv("quiz2_algo.csv", header= True,inferSchema=True)
header = avg_calc.first()
no_header = avg_calc.subtract(header)
no_header

avg_calc contains 2 columns and I am trying to remove the 1st row from both columns, however I am receiving the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-50-24671d91e691> in <module>()
----> 1 no_header = avg_calc.subtract(header)

C:\spark\spark-2.3.0-bin-hadoop2.7\python\pyspark\sql\dataframe.pyc in subtract(self, other)
   1391 
   1392         """
-> 1393         return DataFrame(getattr(self._jdf, "except")(other._jdf), self.sql_ctx)
   1394 
   1395     @since(1.4)

C:\spark\spark-2.3.0-bin-hadoop2.7\python\pyspark\sql\types.pyc in __getattr__(self, item)
   1559             raise AttributeError(item)
   1560         except ValueError:
-> 1561             raise AttributeError(item)
   1562 
   1563     def __setattr__(self, key, value):

AttributeError: _jdf

If anyone can help I would appreciate it!

enter image description here
Example of the data: avg_calc.show(5)

Upvotes: 0

Views: 1285

Answers (1)

Jesse Vogt
Jesse Vogt

Reputation: 16549

first() returns a Row object rather than a DataFrame which is required by subtract. See http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrame.first

You could try something like:

avg_calc.subtract(avg_calc.limit(1))

For example:

>>> df = spark.createDataFrame([Row(x=1), Row(x=2)])
>>> print(df.subtract(df.limit(1)).toPandas())
   x
0  2

Apply an ordering to you dataframe to ensure the row you would like dropped is in the correct location:

>>> from pyspark.sql import functions as F
>>> df = df.orderBy(F.col('CS202 Quiz#2').desc())
>>> df = df.subtract(df.limit(1))

Upvotes: 2

Related Questions