CODEWITHSUNDEEP
streamsets
Yogesh
Yogesh

Reputation: 11

Streamsets job logs to S3

I am looking for solution where we can store all logs (Info, Debug etc) of a Streamsets pipeline (Job) to S3 buckets ?

Currently logs are only available at log console of Streamsets UI only

Upvotes: 0

Views: 80

Answers (2)

Roman
Roman

Reputation: 250

Logs are stored in $SDC_HOME/log directory on the VM running SDC. from there you can copy them to S3. Or you can configure your own path, and map it to an S3 bucket on OS level.

Upvotes: 0

shaine
shaine

Reputation: 549

Looking at the code streamsets uses log4j for all its logging, so you could use something like https://github.com/parvanov/log4j-s3 and create a log write appender.

an alternative is that you could create a new pipeline to consume the logs and write to s3.

Upvotes: 0

Related Questions