CODEWITHSUNDEEP
amazon-s3yamlairflowamazon-redshiftsharepoint-list
user27495465
user27495465

Reputation: 1

Data Ingestion from Amazon Redshift To Sharepoint List

I'm currently working on a project at my company to ingest data from Amazon Redshift to Sharepoint Lists. Some background information: this data flow was initially automated using Azure Data Factory ETL (which is now decomissioned). I was thinking of using PowerAutomate to do so, but my client seeks an alternative way of transforming data from redshift into Sharepoint.

Usually, within my data engineering team, for our end-to-end workflow, we utilise a YAML file to specify where the data comes from, and the jobs on how the data will move from source location/ source file into Amazon S3 (data lake) stored as a parquet file. From here, We'll then move the data from S3 into Amazon Redshift db, and these steps are orchestrated using Airflow.

There's been instances of ingesting data from Sharepoint lists into Airflow, but this is the first time I've had a request of moving data from Redshift into Sharepoint List.

I believe the client seeks a way of this new data flow created using our current tools, S3, Redshift, Airflow to set this job, so that we'll be the ones monitoring the data flow.

Any ideas/ help would be appreciated.

I'm a little bit stumped here because most of the solutions I've found online recommend using PowerAutomate instead.

Upvotes: 0

Views: 61

Answers (0)

Related Questions