CODEWITHSUNDEEP
amazon-web-servicesgoogle-cloud-platformairflow
John
John

Reputation: 165

AWS and GCP centrally managed airflows and Dataflow equivalent for AWS

I have two questions to ask:

  1. So my company has 2 instances of airflow running, one on a GCP provisioned cluster and another on a AWS provisioned cluster. Since GCP has Composer, which helps you to manage airflow, is there a way to sort of integrate the airflow DAGs on the AWS cluster to be managed by GCP as well?

  2. For Batch ETL/Streaming jobs(in python), GCP has Dataflow (Apache Beam) for that. What's the AWS equivalent of that?

Thanks!

Upvotes: 1

Views: 2572

Answers (1)

Duy Nguyen
Duy Nguyen

Reputation: 1015

  1. No, you can't do it, till now you have to use AWS, provision it and manage by yourself. There are some options you can choose: EC2, ECS + Fargate, EKS

  2. Dataflow is equivalent to Amazon Elastic MapReduce (EMR) or AWS Batch Dataflow. Moreover if you want to run current Apache Beam jobs, you can provision Apache Beam in EMR and everything should be the same

Upvotes: 1

Related Questions