CODEWITHSUNDEEP
pythonetlluigi
djWann
djWann

Reputation: 2117

How to make a Luigi task generate an in-memory list as target

I'm trying to write an etl pipeline using luigi. As far as I understand from the documentation a task in luigi can generate a target that can be either some type of file storage or a database. To decrese the processing time I would like to have as an output an in-memory list. Is this possible? Do I have to create a custom target?

Upvotes: 7

Views: 3286

Answers (1)

djWann
djWann

Reputation: 2117

I found out I can use a MockFile as a target. A good example:

http://gouthamanbalaraman.com/blog/building-luigi-task-pipeline.html

Upvotes: 9

Related Questions