CODEWITHSUNDEEP
pythonpython-3.xdataframepython-multiprocessing
Keku
Keku

Reputation: 13

Converting list to dataframe in multiprocessing

I am using multiprocessing in python to improve the performance of the code. Instead of appending the result directly in a dataframe, I am using a common list.

df_list = mp.Manager().list()

The values are calculated and appended in the above list by different processes.

lock.acquire()
    try:
        temp = [a, b, c, d, e, f, g, h, i, j, k, l, m, n]
        df_list.append(temp)
    finally:
        lock.release()

The alphabets are variables containing different values.

Once the child processes terminate, the list contains the following values:

print(df_list)
[[0, '2009-10-01', '2010-01-01', 3, 3, 1, 0.25, 0.9, 6, 100000, 101866.70524177742, 7.68, 100000, 0.0], 
 [1, '2009-10-01', '2010-01-01', 5, 7, 2, 0.25, 0.4, 8, 100000, 101866.70524177742, 9.32, 100000, 0.0]]

Now, I need to convert the list to dataframe.

cols = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N"]
df = pd.DataFrame(df_list, columns = cols)

However, I get the following error:

ValueError                                Traceback (most recent call last)
~\Desktop\Temp\Python\calculate.py in <module>
    469     print(df_list)
    470
--> 471     df = pd.DataFrame(df_list, columns = cols)
    472

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data, index, columns, dtype, copy)
    466                                    dtype=values.dtype, copy=False)
    467             else:
--> 468                 raise ValueError('DataFrame constructor not properly called!')
    469
    470         NDFrame.__init__(self, mgr, fastpath=True)

Could you please let me know if I am doing anything wrong.

EDIT: If we replace the list with standard list it works (as mentioned in the comments as well). However, in multiprocessing we need a manager list to share across the processes.

Also, I found a workaround and it seems to work:

simple_list = []
for l in df_list:
    simple_list.append(l)
df = pd.DataFrame(simple_list, columns = cols)

However, as you can see not a very elegant way of doing things. Still, the question persists - Why can't I convert the manager list to dataframe in a single step. May be I am doing something wrong?

Upvotes: 1

Views: 898

Answers (2)

Roland Smith
Roland Smith

Reputation: 43553

It works fine here (when I use a standard list):

In [1]: import pandas as pd                                                                              

In [2]: df_list = [[0, '2009-10-01', '2010-01-01', 3, 3, 1, 0.25, 0.9, 6, 100000, 101866.70524177742, 7.6
   ...: 8, 100000, 0.0],  
   ...:  [1, '2009-10-01', '2010-01-01', 5, 7, 2, 0.25, 0.4, 8, 100000, 101866.70524177742, 9.32, 100000,
   ...:  0.0]]

In [3]: cols = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N"]

In [4]: df = pd.DataFrame(df_list, columns = cols)                                                       
Out[4]: 
   A           B           C  D  E  F     G    H  I       J              K     L       M    N
0  0  2009-10-01  2010-01-01  3  3  1  0.25  0.9  6  100000  101866.705242  7.68  100000  0.0
1  1  2009-10-01  2010-01-01  5  7  2  0.25  0.4  8  100000  101866.705242  9.32  100000  0.0

So the problem lies with using a proxy. The solution then is to convert it to a list before using it to create a DataFrame. Observe:

In [1]: import multiprocessing as mp

In [2]: proxy = mp.Manager().list([1,2,3])
Out[2]: <ListProxy object, typeid 'list' at 0x8042e5710>

In [3]: reallist = list(proxy)
Out[3]: [1, 2, 3]

In [4]: type(reallist)
Out[4]: list

After the explicit conversion, we're back to a regular old list.

Upvotes: 1

Sukanya Pai
Sukanya Pai

Reputation: 824

I know the below approach for correct conversion from list to Dataframe (there might be many more approaches):

  1. Giving column name as part of the list as below: (I took only the first 2 colums from your set)
    df_list= [[ {"A":0 }, {"B": '2009-10-01'}],
        [ {"A":1}, {"B":'2009-10-01'}]]
  1. Then using the below code, you can convert to a dataframe:
    df = pd.DataFrame(df_list)

Upvotes: 0

Related Questions