CODEWITHSUNDEEP
ssis
Dinesh
Dinesh

Reputation: 303

How read millions of records from flat file?

In my data flow i read data from flat file and insert to oledb destination. In between steps i have used some other components like (lookp, derived, scrip component). I executed my package at time 3500 records read from flat file to oledb destination . so my question is how improve this reading seques 3500 to 100000 this count. one more question is any option available to allocate ram usage in ssis?

Upvotes: 1

Views: 281

Answers (1)

Diego
Diego

Reputation: 36126

I couldn't understand what you mean by "so my question is how improve this reading seques 3500 to 100000 this count". Can you be more clear?

Regarding memory, you can (and should) configure how the cache works on your lookup component. You can set it to none, full or partial.

  • Full: runs the lookup query once
  • Partial: runs the lookup query once per ID and keeps the value
  • None: runs the lookup query every time. It keeps the last value in memory so if two equal values exist in 2 subsequent rows, it won’t run again

and on full and partial you can configure the amount of memory used for the cache

Upvotes: 2

Related Questions