CODEWITHSUNDEEP
pythonpandasscipyout-of-memory
edyvedy13
edyvedy13

Reputation: 2296

python alternative solution scipy spatial distance, current solution returns MemoryError

I had a data frame and function like that:

df = pandas.DataFrame({'Car' : ['BMW_1', 'BMW_2', 'BMW_3', 'WW_1','WW_2','Fiat_1', 'Fiat_2'],
                       'distance'   : [10,25,22,24,37,33,49]})

def my_func(x,y):
   z = 2x + 3y
   return z

I wanted to get pairwise combinations of the distances covered by the cars and use them in my_func. But there are two conditions are that x and y can not be same brands and combinations should not be duplicated. Desired output was something like this:

Car      Distance   Combinations                                
0  BMW_1   10         (BMW_1,WW_1),(BMW_1,WW_2),(BMW_1,Fiat_1),(BMW_1,Fiat_1)
1  BMW_2   25         (BMW_2,WW_1),(BMW_2,WW_2),(BMW_2,Fiat_1),(BMW_2,Fiat_1)
2  BMW_3   22         (BMW_3,WW_1),(BMW_3,WW_2),(BMW_3,Fiat_1),(BMW_3,Fiat_1)
3  WW_1    24         (WW_1, Fiat_1),(WW_1, Fiat_2)
4  WW_2    37         (WW_2, Fiat_1),(WW_2, Fiat_2)
5  Fiat_1  33         None
6  Fiat_2  49         None

//Output
[120, 134, 156, 178]
[113, 145, 134, 132]
[114, 123, 145, 182]
[153, 123] 
[120, 134] 
None 
None

Next Step I wanted to get maximum numbers from the arrays of 'output' row for each brand. And the final data should look like

  Car  Max_Distance
0 BMW  178
1 WW   153
2 Fiat None

MaxU provided me an exccellent answer here:python pandas, a function will be applied to the combinations of the elements in one row based on a condition on the other row

But I keep getting memoryerror although I run my code in super computer since my dataset is extremely large. Is there any more memort efficient way of achieving that? Maybe saving combinations to a database then get the maximums?

Upvotes: 1

Views: 72

Answers (1)

zipa
zipa

Reputation: 27869

So here is the code for the 1st thing:

import pandas as pd
import itertools as it

df = pd.DataFrame({'Car' : ['BMW_1', 'BMW_2', 'BMW_3', 'WW_1','WW_2','Fiat_1', 'Fiat_2'],
                       'Distance'   : [10,25,22,24,37,33,49]})


cars = df['Car'].tolist()
combos = [a for a in list(it.combinations(cars,2)) if a[0].split('_')[0] != a[1].split('_')[0]]

maps_combos = {car: [combo for combo in combos if combo[0] == car] for car in cars}
values = {k:v for k,v in df[['Car', 'Distance']].as_matrix()}
maps_values = {i: [2*value[0] + 3*value[1] for value in j] for i, j in {k: [map(lambda x: values[x], item) for item in v] for k, v in maps_combos.items()}.items() if j}

df['Combinations'] = df['Car'].map(maps_combos)
df['Output'] = df['Car'].map(maps_values)

As for the max, I will need to take a break :)

P.S. I'm not sure that I got the right function for distance multiplication.

EDIT

This max thing (it surely can be done better):

df['Max'] = df['Output'].fillna(0).apply(lambda x: max(x) if x != 0 else np.nan)
df['Brand'] = df['Car'].apply(lambda x: x.split('_')[0])
brand_max = df[['Brand', 'Max']].groupby('Brand').max()

Upvotes: 1

Related Questions