CODEWITHSUNDEEP
azure-storagesnowflake-cloud-data-platformoracle-cloud-infrastructure
Harman
Harman

Reputation: 11

Oracle Cloud to Azure Cloud storage

We have a requirement to move data from oracle Cloud storage to Azure Cloud storage.

The requirement is basically to move data from an Oracle ADW database (hosted on Oracle cloud) to Snowflake database (hosted on Azure).

Since the data volume in tables is huge (some with 60mil+ records) we do not wish to use any ETL tool and instead want to setup a pipeline as below.

Oracle ADW database -> Store data in Oracle storage --> Move data to Azure Cloud storage -> Load into Snowflake using snowpipe or similar snowflake utilities.

How should I go about this implementation?

Also share your views on whether we can use Oracle fastconnect and Azure ExpressRoute to directly pull data from Oracle Cloud onto snowflake (or into Azure storage)

Upvotes: 1

Views: 416

Answers (1)

Gavin Powell
Gavin Powell

Reputation: 1

I am looking for the same thing with the simplest method from Oracle (on prem but could be cloud), into Snowflake. Looks like data must be exporeted or dropped to external tables, shifted to Azure Blob storage (like AWS S3), then pushed into Snowflake using COPY INTO - basically copying on disk external tables. This is what Snowpipe does:

"Snowpipe copies the files into a queue, from which they are loaded into the target table in a continuous, serverless fashion based on parameters defined in a specified pipe object. The following table indicates the cloud storage service support for automated Snowpipe from Snowflake accounts hosted on each cloud platform:"

It's been a while since I have worked with this. The other option is GoldenGate, which was not expensive the last time I looked into it:

https://www.snowflake.com/blog/continuous-data-replication-into-snowflake-with-oracle-goldengate/

Easy, simple, fast. Anyone have any better ideas would be appreciated.

Upvotes: 0

Related Questions