CODEWITHSUNDEEP
amazon-web-servicesairflowamazon-redshiftamazon-vpc
luke33
luke33

Reputation: 33

Can not connect to Redshift from Airflow - ConnectionRefusedError

I am trying to create a connection to AWS Redshift from the Airflow Web UI, but so far it does not work. I always get the error: ('communication error', ConnectionRefusedError(111, 'Connection refused').

My Redshift is in a subnet of a VPC, but I have allowed public access. In its security group I have furthermore added: 0.0.0.0/0 IPV4 incoming access and my own IP as incoming access allowed.

As The input parameters to the airflow connection template, I use:

As the connection-id: An arbitrary name

host: redshift-clusterxxxxxxxxxxxxxxxxxxxxxxxxxx.redshift.amazonaws.com The other parameters are also correct.

I have build a little python using redshift_connector to test whether I can connect from there but I am always getting a timeout error.

Has anyone got further ideas on how to solve this issue?

Upvotes: 0

Views: 992

Answers (2)

mata94
mata94

Reputation: 1

f you are still experiencing connection issues with RedshiftSQLOperator, you should check your AWS Security Group settings to ensure your Redshift Serverless instance is accessible. Here's how you can do it:

Go to AWS Console → VPC → Security Groups. Find the Security Group attached to Redshift Serverless: Go to Redshift Serverless → Workgroup → Networking & Security. Note down the Security Group ID (e.g., sg-xxxxxxx). Modify the Security Group inbound rules: Go to "Inbound Rules" → Edit rules. Add a new rule: Type: Redshift Protocol: TCP Port: 5439 (default Redshift Serverless port) Source: If you are on a personal computer, add your public IP If you want to allow all connections (for testing only), use 0.0.0.0/0. Click "Save changes". If this rule was missing, it was likely the cause of your issue!

I’m a bit late to the party, but I hope this helps someone who might still be pulling their hair out over this issue.

Upvotes: 0

Ricardo Sueiras
Ricardo Sueiras

Reputation: 301

I have run into this same issue, and ended up switching to a different Airflow Operator which solved the issue for me.

I had been using from airflow.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperator and I switched to from airflow.providers.amazon.aws.operators.redshift_data import RedshiftDataOperator and this then magically worked.

I was using the amazon-provider package 6.0.0 and MWAA version 2.2.2 and 2.4.3.

I was determined to try and figure out why the other Operator did not work but ran out of time. I did find this link https://repost.aws/knowledge-center/cannot-connect-redshift-cluster and was going to setup the second option as I think this fits my situation but never got around to doing that.

Upvotes: 0

Related Questions