How to intercept Spark SQL queries

Question

I am working on modifying the Apache Spark source code to capture SQL queries executed using spark.sql() at runtime, modify them (if needed), and then execute the modified SQL.

Requirements:

• The SQL query should be captured at runtime (not from logs, lineage, or physical/logical plans).
• The query should be printed to the console and modified dynamically before execution.
• No additional code should be required from the client-side—users should be able to run spark.sql() as usual, and the system should automatically handle query capturing and modification.

I tried modifying several listener classes, but they mostly display logical/physical plans instead of raw SQL. Files I Have Modified (But Not Working as Expected)

1. SparkFirehoseListener.java
2. SQLListener.scala
3. QueryExecutionListener.scala

These files mostly log query plans rather than capturing the raw SQL at runtime.

Answer 1

It seems to me that you might need to extend this Abstract Scala class