CODEWITHSUNDEEP
csvamazon-s3rubygemsetlkiba-etl
Mohammed Attaullah Khan
Mohammed Attaullah Khan

Reputation: 11

Kiba-etl Mutliple Transformation-Multiple Destination

I am trying to having mutliple transformation ,distributing it to multiple destination . for example : 

orginal.csv: 
title 
movies1
movies2
movies3
movies4

adding to the .themoviedb and it is transformed to this 

better_movies.csv:
title
movies1=9
movies2=5.5
movies3=7
movies4=8

Again i want to transform and push this file to s3 rated_movies.csv: movies1=9 movies=8

this should be the output how can i do this and then push it to my s3 folder ?

Upvotes: 1

Views: 212

Answers (1)

Thibaut Barrère
Thibaut Barrère

Reputation: 8873

Based from your question title it's not clear exactly what you are asking for, could you clarify?

If we imagine that you have a source file (CSV) which is transformed by your transforms, then written as a target file (CSV again), which you'd like to send to S3, one way would be to use a post_process block to act on the target file once it's ready.

Roughly it would give :

source CSVSource, file: my_source_file

transform
transform
transform

destination CSVDestination, file: my_target_file

post_process do
  # send your file to s3, by shelling out to `aws s3 cp`
  # (but make sure to check the exit code if you do so!)
  # or by using the AWS ruby SDK
end

There are of course many other possibilities, such as adding a new destination below the first one.

Hope this helps!

Upvotes: 1

Related Questions