CODEWITHSUNDEEP
springgitspring-bootspring-cloud-dataflow
Chaos
Chaos

Reputation: 471

Spring Cloud Data Flow Git source

I'm interested in watching a git repository for changes and then acting on the files of that repository. I could set a webhook in github to notify Data Flow when the repository changes, but I need to download/clone the files to process them. Is there some local storage that is guaranteed to deployments where I could do something like that?

Basically, is there local storage available to Processors in the Data Flow deployment pipeline so that they can save files to disk and process them to the next stage of the pipeline?

Thanks!

Upvotes: 0

Views: 172

Answers (1)

Sabby Anandan
Sabby Anandan

Reputation: 5651

Spring Cloud Data Flow is nothing but a Spring Boot application. SCDF doesn't interact directly with the Apps, message brokers, or the underlying platform components, including the volume mounts.

That said, if you need your applications to download and process the downloaded files, you'd have to rely on the components of where the application is running.

For example, when using SCDF in Local-mode, all the apps are running as Java process in the machine/VM, so if the processor has access to the local file-system, then yes, you could refer to it as file:/// resource. You could use the SFTP to JDBC data pipeline as a reference. Here, the SFTP source locally downloads the file, reads the metadata of the file, and then sends the metadata as the payload to the downstream app, which in turn launches a Task to write the content of the file to a database.

If you're running SCDF in Cloud Foundry, you will rely on managed services like PCF Volume Services, Minio, and similar things. (See docs)

Likewise, with SCDF on Kubernetes, you could use the K8s Volume Mounts with your choice of backend.(See docs)

Upvotes: 1

Related Questions