CODEWITHSUNDEEP
pythonnumpypandas
member555
member555

Reputation: 807

pandas read_csv() and python iterator as input

I want to use pandas read_csv() func where the input is a python iterator, where each next() will bring to the next line of my text file. What would you suggest me to do? I want best performance.
As I understood, StringIO works in that case, but i would rather to not use that.

BTW, after that I'm using as_matrix() function in order to create a numpy array.
Doing so because it's much faster than np.loadtxt() func which is horribly slow :(

Upvotes: 1

Views: 1183

Answers (1)

member555
member555

Reputation: 807

You should use :

 from io import StringIO
 pd.read_csv(StringIO("\n".join(iter)))

where iter is your iterator / generator variable.
This is still faster than using np.loadtxt(iter)

Upvotes: 2

Related Questions