JFerro
JFerro

Reputation: 3433

Python Vaex read txt file parse it and append it

I am dealing with text files containing space separated values

File1.txt: text1 text2 text3 text4 ...

I would like to read the file into a vaex df and concatenate it to a so called TOTALDF vaex df.

I want to use vaex and not pandas because the df is so big that it does not fit in memory

I posted another question related to the issue here: python: How concatenate pandas dataframes with VAEX

So I have a partial solution (workaroind). I.e. creating hdf5 dataframes. But that takes time. Too much time. The question is if I can concatenate one by one a vaex df read from a csv file into this total one.

Upvotes: 0

Views: 215

Answers (0)

Related Questions