CODEWITHSUNDEEP
pythonpython-2.7unit-testingpysparkapache-spark-sql
Nevermore
Nevermore

Reputation: 7389

Check if two pyspark Rows are equal

I am writing unit tests for a Spark job, and some of the outputs are named tuples: pyspark.sql.Row

How can I assert their equality?

actual = get_data(df)
expected = Row(total=4, unique_ids=2)
self.assertEqual(actual, expected)

When I do this, the values are rearranged in an order I can not determine.

Upvotes: 0

Views: 2216

Answers (1)

pault
pault

Reputation: 43494

Your code should work as written because according to the docs:

the fields will be sorted by names.

Nevertheless, another way is to use the asDict() method of the pySpark.sql.Row and compare them as dictionaries:

actual = get_data(df)
expected = Row(total=4, unique_ids=2)
self.assertEqual(actual.asDict(), expected.asDict())

Upvotes: 1

Related Questions