Geospark IllegalArgumentException: Number of partitions must be >= 0

Question

I'm trying to run a simple intersect on a couple of tables with geometries and get this error.

IllegalArgumentException: Number of partitions must be >= 0

My script.

tableA.
join(tableB, expr("ST_Intersects(geom, point)")).
show

This is table A. It has a few million rows.

spark.table("ta").
withColumn("point", expr("ST_Point(CAST(lon AS Decimal(24,20)), CAST(lat AS Decimal(24,20)))"))

And the result.

+-----------+-----------+--------------------+
|        lat|        lon|               point|
+-----------+-----------+--------------------+
|  44.978577|  30.172431|POINT (30.172431 ...|
|  44.707343|  30.794019|POINT (30.794019 ...|
|  44.817301|  30.704576|POINT (30.704576 ...|
|  44.710767|  30.657547|POINT (30.657547 ...|
|   44.88699|  30.521111|POINT (30.521111 ...|
|     44.779|    30.6296|POINT (30.6296 55...|
|  44.653987|  30.572032|POINT (30.572032 ...|
|  44.763931|  30.601646|POINT (30.601646 ...|
|44.44440079|30.50870132|POINT (30.5087013...|
|  44.707493|  30.575095|POINT (30.575095 ...|
|  44.566665|   30.56598|POINT (30.56598 5...|
|   44.58322|  30.209977|POINT (30.209977 ...|
|  44.687525|  30.665842|POINT (30.665842 ...|
|44.90000153|30.62870026|POINT (30.6287002...|
|   44.85094|  30.560021|POINT (30.560021 ...|
|   44.83429|   30.49514|POINT (30.49514 5...|
|  44.740523|  30.890627|POINT (30.890627 ...|
|  44.544804|  30.328373|POINT (30.328373 ...|
|   44.46986|    30.5456|POINT (30.5456 55...|
|    44.8912|    30.6089|POINT (30.6089 55...|
+-----------+-----------+--------------------+

This is table B. It has only 1 row.

spark.table("tb").
withColumn("geom", expr("ST_GeomFromWKT(wkt)"))

And what show gives me.

+--------------------+--------------------+
|                 wkt|                geom|
+--------------------+--------------------+
|MULTIPOLYGON (((3...|MULTIPOLYGON (((3...|
+--------------------+--------------------+

What's with this error? How do I fix it?

Answer 1

I had the order wrong. According to the docs it's.

boolean ST_Intersects( geometry geomA , geometry geomB )

Changing to expr("ST_Intersects(point, geom)") solved it.