CODEWITHSUNDEEP
apache-sparkapache-calcite
Sathish
Sathish

Reputation: 93

Apache Calcite Data Federation Usecase

Just want to check if the Apache Calcite can be used for the use case "Data Federation"(query with multiple databases).

The idea is I have a master query (5 tables) that has tables from one database (say Hive) and 3 tables from another database (say MySQL).

I read from Calcite documentation that it can push down Join and GroupBy but I could not understand it? Can anyone help me understand this?

Upvotes: 2

Views: 1535

Answers (1)

zinking
zinking

Reputation: 5695

I will try to answer. you can as well send questions to the mailing list. [email protected] you are more likely get answer there.

Can I execute master query on multiple database from one JDBC Client interface ? If this is possible; where the query execution (particularly inter database join) happens?

yes, you can. the Inter database join happens in your memory where calcite runs.

Can I get a physical plan from Calcite where I can execute explicitly in another execution engine?

yes, you can. a lot of calcite consumers are doing this way. but you will have to wrap around the calcite rule system, I mean excute

I read from calcite documentation that it can push down Join and GroupBy but I could not understand it? Can anyone help me understand this?

these are the SQL optimisations that the engine does. imagine a groupBy which could have happened on a tiny table but actually specified after joining with a huge table.

Upvotes: 4

Related Questions