CODEWITHSUNDEEP
pythonpandasdictionarycombinations
Nico
Nico

Reputation: 311

Fill a list/pandas.dataframe with all the missing data combinations (like complete() in R)

I have data set like the following (this is an example, it actually has 66k rows):

        Type       Food      Loc  Num
0      Fruit     Banana  House-1   15
1      Fruit     Banana  House-2    4
2      Fruit      Apple  House-2    6
3      Fruit      Apple  House-3    8
4  Vegetable   Broccoli  House-3    8
5  Vegetable    Lettuce  House-4   12
6  Vegetable    Peppers  House-5    3
7  Vegetable       Corn  House-4    4
8  Seasoning  Olive Oil  House-6    2
9  Seasoning    Vinegar  House-7    2

I'd like to fill all the missing combinations (how many bananas are there in Houses 3-7?, how many peppers are there elsewhere than House-5?) with 0, to get something like:

        Type       Food      Loc  Num
0      Fruit     Banana  House-1   15
1      Fruit     Banana  House-2    4
2      Fruit     Banana  House-3    0
... fill remaining houses with zeros
6      Fruit     Banana  House-7    0
7      Fruit      Apple  House-1    0
8      Fruit      Apple  House-2    6
9      Fruit      Apple  House-3    8
... fill remaining houses with zeros
14  Vegetable   Broccoli  House-1    0
15  Vegetable   Broccoli  House-2    0
16  Vegetable   Broccoli  House-3    8
... etc    
n   Seasoning    Vinegar  House-7    2

I know that R has the complete function integrated.

Right now I had been working with a list that was digested from the original DataFrame, which I transformed into a dictionary.

for key,grp in fruit.groupby(level=0):
        dir[key] = test.ix[key].values.tolist()

fruit = {'Banana': [[1.0,15.0], [2.0,4.0],
         'Apple': [[2.0,6.0], [3.0,8.0]

#Type = {fruit1:[[Loc1,Count1],...,[Locn],[Countn],
#... fruitn:[...]}

I designed this function to apply to the assignation rule of the dictionary:

def fill_zeros(list):
    final = [0] * 127
    for i in list:
        final[int(i[0])] = i[1]
    return final

That works on individual "fruits":

print fill_zeros(test.ix['QLLSEEEKK'].values.tolist())
print fill_zeros(test.ix['GAVPLEMLEIALR'].values.tolist())
print fill_zeros(test.ix['VPVNLLNSPDCDVK'].values.tolist())

But doesn't on the dictionary:

for key,grp in test.groupby(level=0):
        dir[key] = fill_zeros(test.ix[key].values.tolist())

Traceback (most recent call last):
  File "peptidecount.py", line 59, in <module>
    print fill_zeros(test.ix[str(key)].values.tolist())
  File "peptidecount.py", line 43, in fill_zeros
    final[int(i[0])] = i[1]
TypeError: 'float' object has no attribute '__getitem__'

Apparently I'm not iterating correctly on the dictionary. Is there a way to correct that? Or is there a more suitable function to apply directly on the DataFrame?

Upvotes: 4

Views: 977

Answers (2)

TomAugspurger
TomAugspurger

Reputation: 28946

You could use a reindex.

First you'll need a list of the valid (type, food) pairs. I'll get it from the data itself, rather than writing them out.

In [88]: kinds = list(df[['Type', 'Food']].drop_duplicates().itertuples(index=False))

In [89]: kinds
Out[89]:
[('Fruit', 'Banana'),
 ('Fruit', 'Apple'),
 ('Vegetable', 'Broccoli'),
 ('Vegetable', 'Lettuce'),
 ('Vegetable', 'Peppers'),
 ('Vegetable', 'Corn'),
 ('Seasoning', 'Olive Oil'),
 ('Seasoning', 'Vinegar')]

Now we'll generate all the pairs for those kinds with the houses using itertools.product.

In [93]: from itertools import product

In [94]: houses = ['House-%s' % x for x in range(1, 8)]

In [95]: idx = [(x.Type, x.Food, house) for x, house in product(kinds, houses)]

In [96]: idx[:2]
Out[96]: [('Fruit', 'Banana', 'House-1'), ('Fruit', 'Banana', 'House-2')]

And now you can use set_index and reindex to get the missing observations.

In [98]: df.set_index(['Type', 'Food', 'Loc']).reindex(idx, fill_value=0)
Out[98]:
                           Num
Type      Food    Loc
Fruit     Banana  House-1   15
                  House-2    4
                  House-3    0
                  House-4    0
                  House-5    0
...                        ...
Seasoning Vinegar House-3    0
                  House-4    0
                  House-5    0
                  House-6    0
                  House-7    2

[56 rows x 1 columns]

Upvotes: 2

piRSquared
piRSquared

Reputation: 294508

This should work:

cond0 = df.Num.isnull()
cond1 = df.Food == 'Banana'
cond2 = df.Loc.str.match(r'House-[34567]')
cond3 = df.Food == 'Peppers'
cond4 = df.Loc != 'House-5'

missing_bananas = cond0 & cond1 & cond2
missing_peppers = cond0 & cond3 & cond4
missing_food = missing_bananas | missing_peppers

df.loc[missing_food] = df.loc[missing_food].fillna(0)

Upvotes: 0

Related Questions