CODEWITHSUNDEEP
postgresqlapache-sparkapache-spark-sql
yyuankm
yyuankm

Reputation: 365

How does Spark SQL execute SQL query with joining operation?

For the following operation to run sql statement in spark sql to join two tables in PostgreSQL:

val df = spark.read.jdbc(url, 'select * from table_1 join table_2 on a where x', connproperties);

Will Database Engine execute the joining operation and sends back the joined results? Or will the Database send all records of table_1 and table_2 to spark job and spark job do the joining? Are there some documentation to explain this operation? Thanks!

Upvotes: 0

Views: 647

Answers (1)

Frank Heikens
Frank Heikens

Reputation: 127596

The PostgreSQL database will only return a single resultset from a single query. If you would use valid SQL, that could be the joined result. Or nothing, in case no records match your conditions.

Upvotes: 1

Related Questions