CODEWITHSUNDEEP
pythonkedro
Waylon Walker
Waylon Walker

Reputation: 563

How to load kedro DataSet object dynamically

I am currently using the yaml api to create all of my datasets with kedro==15.5. I would like to be able to peer into this information from time to time dynamically. It appears that I can get to this information with the io.datasets which is a _FrozenDatasets object. I cannot loop over it or access it programatically though.

Specific Use Case

Specifically I would like to add a test that loops over the datasets to check that there are not multiple catalog entries using the same filepath. Is this possible without using eval? Currently I think would need to do something like this 

filepaths = {}
for entry_name in io.list()
   eval(f'filepaths[{entry_name}] = io.datasets.{entry_name}'.filepath)

Upvotes: 1

Views: 772

Answers (1)

dres
dres

Reputation: 1211

Unfortunately, I don't think AbstractDataSet (from which they are all defined) has a property for filepath or the config that built it. You can read the ProjectContext config but that won't cover datasets that were dynamically built.

Upvotes: 0

Related Questions