CODEWITHSUNDEEP
sftpazure-synapse
Jing
Jing

Reputation: 13

How to download files from SFTP that doesn't have actural files in Synapse?

I have a little bit complicate situation here: I need to download files from a SFTP daily. I connect to the SFTP with username and SSH key, the keys have a passphrase. This SFTP has no actual files. All the files on the server is 0 bytes. The server will dynamicly generate the file if it get a "get" command.

So when I connect the SFTP with Winscp, everything went perfectly. But I have to do it in Synapse.

I managed to connect it in Pipeline with copy activity, and I managed to download all the files, but with no data content inside.

Does anyone know how I can download the files with content?

Upvotes: 0

Views: 246

Answers (1)

Kranthi Pakala
Kranthi Pakala

Reputation: 1422

If you actually have files with content in SFTP location, then they should also be automatically copied using the pipeline in your Synapse. In case if you just want to copy the files that are having the content and ignore empty files, then you will have to use a get metadata activity to check the size of the file (i.e., > 0 bytes) and then filter those files only to copy to your desired destination. Using the childItems you can get the fileName, Type and Size and use these properties in the subsequent copy activity to only copy filter files to your destination.

Upvotes: 0

Related Questions