CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
NYCeyes
NYCeyes

Reputation: 5659

Apache Spark 2.0: Expression-string to orderBy() / sort() column in descending order

I'm looking at a book example similar to the following (virtually identical):

>>> from pyspark.sql import functions as sFn
>>>   # Note: I import Spark functions this way to avoid name collisions w/ Python.
>>>   # Usage below: sFn.expr(), sFn.col(), etc.

>>> col0 = [0, 1, 2, 3]
>>> col1 = [4, 5, 6, 7]

>>> myDF = spark.createDataFrame(zip(col0, col1),
                                 schema=['col0', 'col1'])
>>> print(myDF)
>>> myDF.show()
>>> myDF.orderBy(sFn.expr('col0 desc')).show() # <--- Problem line. Doesn't descend.

Now the book example claims that the last statement would order by col0 in descending fashion, but it does not:

DataFrame[col0: bigint, col1: bigint]

+----+----+
|col0|col1|
+----+----+
|   0|   4|
|   1|   5|
|   2|   6|
|   3|   7|
+----+----+

+----+----+
|col0|col1|
+----+----+
|   0|   4|
|   1|   5|
|   2|   6|
|   3|   7|
+----+----+

This syntax variant, however, has always worked for me:

myDF.orderBy(sFn.col("col0").desc()).show()

Is the problematic variation above a typo or errata? And if it is a typo or errata, what tweak is necessary to make it work?

Thank you.

Upvotes: 2

Views: 885

Answers (1)

akuiper
akuiper

Reputation: 214927

In sFn.expr('col0 desc'), desc is translated as an alias instead of an order by modifier, as you can see by typing it in the console:

sFn.expr('col0 desc')
# Column<col0 AS `desc`>

And here are several other options you can choose from depending on what you need:

 myDF.orderBy('col0', ascending=0).show()
+----+----+
|col0|col1|
+----+----+
|   3|   7|
|   2|   6|
|   1|   5|
|   0|   4|
+----+----+


myDF.orderBy(sFn.desc('col0')).show()
+----+----+
|col0|col1|
+----+----+
|   3|   7|
|   2|   6|
|   1|   5|
|   0|   4|
+----+----+

myDF.orderBy(myDF.col0.desc()).show()
+----+----+
|col0|col1|
+----+----+
|   3|   7|
|   2|   6|
|   1|   5|
|   0|   4|
+----+----+

Upvotes: 4

Related Questions