CODEWITHSUNDEEP
apache-sparkpysparkspark-streamingazure-synapseazure-eventhub
s528060
s528060

Reputation: 83

Consume EventHub messages using spark structured streaming job with Service Principal + Certificate Auth

I want to use structured streaming for EventHub message consumption using pyspark in Azure Synapse, I found out this documentation https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/use-aad-authentication-to-connect-eventhubs.md that requires to create a callback function and load it into the spark pool as a package, has anyone done something similar for pyspark?

enter image description here

Upvotes: 0

Views: 282

Answers (1)

JayashankarGS
JayashankarGS

Reputation: 8140

Even in the pyspark you need to use the callback class. however, you still need to use that java class.

If you are using databricks then you can give direct class name like below, after you installing azure-eventhubs-spark_2.12

"kafkashaded.org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler"

For synapse, you need to build the jar file out that class and add it as a package.

You can use either this project or the one you are currently referring to.

For more details refer this stack solution

Upvotes: 0

Related Questions