Reputation: 3433
I am dealing with text files containing space separated values
File1.txt: text1 text2 text3 text4 ...
I would like to read the file into a vaex df and concatenate it to a so called TOTALDF vaex df.
I want to use vaex and not pandas because the df is so big that it does not fit in memory
I posted another question related to the issue here: python: How concatenate pandas dataframes with VAEX
So I have a partial solution (workaroind). I.e. creating hdf5 dataframes. But that takes time. Too much time. The question is if I can concatenate one by one a vaex df read from a csv file into this total one.
Upvotes: 0
Views: 215