CODEWITHSUNDEEP
pythonpandasmissing-data
Simon B
Simon B

Reputation: 329

Add missing rows based on column

I have given the following df

df = pd.DataFrame(data = {'day': [1, 1, 1, 2, 2, 3], 'pos': 2*[1, 14, 18], 'value': 2*[1, 2, 3]}    
df

    day pos value
0   1   1   1
1   1   14  2
2   1   18  3
3   2   1   1
4   2   14  2
5   3   18  3

and i want to fill in rows such that every day has every possible value of column 'pos'

desired result:

    day pos value
0   1   1   1.0
1   1   14  2.0
2   1   18  3.0
3   2   1   1.0
4   2   14  2.0
5   2   18  NaN
6   3   1   NaN
7   3   14  NaN
8   3   18  3.0

Proposition:

df.set_index('pos').reindex(pd.Index(3*[1,14,18])).reset_index()

yields:

ValueError: cannot reindex from a duplicate axis

Upvotes: 6

Views: 336

Answers (4)

sammywemmy
sammywemmy

Reputation: 28644

You could use the complete function from pyjanitor to expose the missing values :

#  pip install pyjanitor
import pandas as pd
import janitor as jn

df.complete('day', 'pos')
   day  pos  value
0    1    1    1.0
1    1   14    2.0
2    1   18    3.0
3    2    1    1.0
4    2   14    2.0
5    2   18    NaN
6    3    1    NaN
7    3   14    NaN
8    3   18    3.0

Upvotes: 0

rafaelc
rafaelc

Reputation: 59274

I'd avoid the manual product of all possible values.

Instead, one can get the unique values and just reindex per day:

u = df.pos.unique()

df.groupby('day').apply(lambda s: s.set_index('pos').reindex(u))['value']\
  .reset_index()

   day  pos  value
0    1    1    1.0
1    1   14    2.0
2    1   18    3.0
3    2    1    1.0
4    2   14    2.0
5    2   18    NaN
6    3    1    NaN
7    3   14    NaN
8    3   18    3.0

Upvotes: 2

Henry Yik
Henry Yik

Reputation: 22493

You can reindex:

s = pd.MultiIndex.from_product([df["day"].unique(),df["pos"].unique()], names=["day","pos"])

print (df.set_index(["day","pos"]).reindex(s).reset_index())

   day  pos  value
0    1    1    1.0
1    1   14    2.0
2    1   18    3.0
3    2    1    1.0
4    2   14    2.0
5    2   18    NaN
6    3    1    NaN
7    3   14    NaN
8    3   18    3.0

Upvotes: 2

Quang Hoang
Quang Hoang

Reputation: 150735

Let's try pivot then stack:

df.pivot('day','pos','value').stack(dropna=False).reset_index(name='value')

Output:

   day  pos  value
0    1    1    1.0
1    1   14    2.0
2    1   18    3.0
3    2    1    1.0
4    2   14    2.0
5    2   18    NaN
6    3    1    NaN
7    3   14    NaN
8    3   18    3.0

Option 2: merge with MultiIndex:

df.merge(pd.DataFrame(index=pd.MultiIndex.from_product([df['day'].unique(), df['pos'].unique()])),
         left_on=['day','pos'], right_index=True, how='outer')

Output:

   day  pos  value
0    1    1    1.0
1    1   14    2.0
2    1   18    3.0
3    2    1    1.0
4    2   14    2.0
5    3   18    3.0
5    2   18    NaN
5    3    1    NaN
5    3   14    NaN

Upvotes: 4

Related Questions