CODEWITHSUNDEEP
azure-blob-storagesnowflake-cloud-data-platformazure-data-factoryservicenowservicenow-rest-api
MattyKluch
MattyKluch

Reputation: 29

Azure Data Factory: Loading large table from Service now into Blob storage, why won't ADF pull all the records from the table into a json/orc file?

So, I'm pretty new to using Azure Data Factory and I've come across some strange behavior using ADF and Service Now as a source.

Preface: I'm working with a client that uses Service Now and Snowflake. The goal is to import "larger" tables (think 900k rows) from Service Now into snowflake using Blob storage as a intermediary.

The way it's working is that I set up a copy data task to pull the table from Service Now into a json or orc file. Then I use a Copy into command in snowflake to ingest the data into a Snowflake table. Relatively simple.

What I noticed however is that the 'Copy Data' task in ADF is only pulling less than half of the Service now table into the blob storage file. I can't figure out why. The table is roughly 937k records. The ADF job finishes successfully after only loading 412k records. I have no filtering happening in the copy data task. This is straight source to sink (source to target).

Having tested a smaller table (120k records) ADF correctly loads the full number of records into the blob storage json or orc file. I have a great deal of experience with informatica and when loading this same Service Now table in informatica, all of the 937k rows are copied.

Why can't ADF see all 937k rows in this table? It's as if ADF only can see/copy 412k before it says "completed".

Upvotes: 0

Views: 714

Answers (1)

Ugur
Ugur

Reputation: 1

It haas to do with internal pagination of the ADF driver. ServiceNow side experiences a timeout but the ADF does not understand this, and continues with the next page.

Upvotes: 0

Related Questions