CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beam
Sumit Desai
Sumit Desai

Reputation: 1770

Database IAM authentication failing from Google Dataflow instance

I have a Python based application that is using Apache beam in batch mode and Google Dataflow as a worker.

The first step of my pipeline is to read data from a database table as input PCollection. The library that I have used as Input connector requires DB build-in user and password (not IAM user) and first step is getting executed successfully.

Now, in second step, I want to update the DB rows (just 1 right now for testing) to IN_PROGRESS. Here, I am using an IAM user which I am also using outside of Dataflow. But I am getting an error in dataflow pipeline -

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "xx.xx.xxx.xx", port 5432 failed: FATAL: AlloyDB IAM user authentication failed for user "{iam_user}".

I also tried creating a new IAM user corresponding to the service account I am using for workers and provided it with the same permissions as the IAM user outside of dataflow. But ,I am still seeing the same error. From logs, I can see DB IP ,DB name and IAM users are correctly being passed.

Is there anything additional that I should be doing for an IAM user to successfully connect to DB?

Upvotes: 0

Views: 145

Answers (1)

Benjamin Coutellier
Benjamin Coutellier

Reputation: 21

In addition to the roles mentioned in the documentation, the iam.oauthClientViewer role should be added. With these three roles, we were able to connect using IAM authentication from our Dataflow job. An update to the documentation would be appreciated :)

Upvotes: 0

Related Questions