CODEWITHSUNDEEP
pythonpandas
user1581390
user1581390

Reputation: 1998

Pandas: Turn a Column into a Table

I have a table that is actually two tables. Each row in the table is the row from table A and the row from table B, separated by '|'. Columns in each one are separated by ','.

This is a HUGE table (200 GB) so I need this done efficiently.

Sample data:

0.0,0|586,abc,6
0.4,2|416,efg,3
1.0,8|007,hik,1

I want to get table A as a pandas table, and table B as a separate pandas table.

Upvotes: 0

Views: 475

Answers (1)

Jared Wilber
Jared Wilber

Reputation: 6775

"This is a HUGE table (200 GB) so I need this done efficiently." - then don't use pandas.

If you must, you could first read the data in as a single dataframe:

  df = pd.read_csv('test.txt', header=None, sep='[,|]')

Then separate the data as required:

  df1 = df.iloc[:, 0:3]
  df2 = df.iloc[:, 3:5]
  del df

This deals with the separation, and only loads the data once, but you should look into a different tool for something of that size...

Upvotes: 2

Related Questions