CODEWITHSUNDEEP
scalaapache-spark
dedpo
dedpo

Reputation: 502

Why can't I use sort or orderby on DataFrame?

When I try to sort a DataFrame:

val df1 = df.toDF().sort(desc("sourceId"))

I get:

17/11/07 15:15:37 ERROR Executor: Exception in task 3.0 in stage 114.0 (TID 218) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.math.Ordering$$anon$4 Note: To register this class use: kryo.register(scala.math.Ordering$$anon$4.class); Serialization trace: ord (org.apache.spark.util.BoundedPriorityQueue) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21) at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:312) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:364) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: Class is not registered: scala.math.Ordering$$anon$4

I've also tried order by, but neither works.

What is the issue here? Do I have to import ordering scala.math.order?

Upvotes: 0

Views: 337

Answers (1)

user8902690
user8902690

Reputation: 36

Looks like you're using spark.kryo.registrationRequired

spark.kryo.registrationRequired  true

Please either set it to false:

spark.kryo.registrationRequired  false

or add required class to spark.kryo.classesToRegister

Upvotes: 2

Related Questions