CODEWITHSUNDEEP
pythonpandasdataframecount
ctrl_z
ctrl_z

Reputation: 288

Counting the occurence of data from 2 data frames (linked by an index) in Pandas

I have 2 large dataframes (these below are just extracts) containing information about road accidents, where df_veh contains details about vehicles and df_ped contains the count of pedestrians involved in each accident. veh_type shows the type of vehicles involved in an accident (1=bicycle, 2=car, 3=bus). They are linked with acc_index indicating unique accident.

veh_data = {'acc_index':  ['001', '002', '002', '003', '003', '004', '005', '005', '006',
                           '006', '007', '007', '008', '008', '008', '009', '009', '009'],
         'veh_type': ['1', '1', '2', '1', '1', '1', '2', '2', '2', '3', '1', '2', '1', '1', 
                      '1', '1', '2', '2'] }

df_veh = pd.DataFrame (veh_data, columns = ['acc_index', 'veh_type'])       

ped_data = {'acc_index':  ['001', '002', '003', '004', '005', '006', '007', '008', '009'],
            'pedestrians': ['1', '2', '0', '1', '4', '3', '0', '1', '2'] }
df_ped = pd.DataFrame (ped_data, columns = ['acc_index', 'pedestrians'])

What I want to do is to count the number of accidents (BY UNIQUE acc_index ONLY ONCE):

  1. between cars and bicycles (veh_type==1 and veh_type==2)
  2. between bicycles and pedestrians (veh_type==1 and pedestrians>=1)
  3. between cars and pedestrians (veh_type==2 and pedestrians>=1)
  4. between only cars (veh_type==2 for the same acc_index)
  5. between only bicycles (veh_type==1 for the same acc_index)
  6. between only pedestrians (pedestrians>=1 for the same acc_index)

I tried to do it in different ways but in the end, I get different results so I am confused. For example, I tried to count bike-pedestrian accidents like this:

df_bikes = df_veh[df_veh['veh_type']==1].groupby('acc_index').sum().reset_index()
bike_ped = pd.merge(df_bikes, df_ped, how='outer', on='acc_index')
bike_ped[(bike_ped['veh_type']==1) & (bike_ped['pedestrians']>=1)].groupby(
    'acc_index').sum().reset_index()[['acc_index', 'veh_type', 'pedestrians']]

Another example, this is how I counted accidents between cars and bicycles thanks to comments in this post. I believe this one is correct at least. I am trying to find the simplest way to do this (but also displaying the rows counted).

bike_car = df_veh[def_veh.groupby('acc_index')['veh_type'].
                transform(lambda g: not({1, 2} - {*g}))][['acc_index', 'veh_type']]
len(bike_car.groupby(['acc_index']).size().reset_index()))

Upvotes: 0

Views: 60

Answers (1)

Parfait
Parfait

Reputation: 107567

Consider pivoting the vehicle data with pivot_table that is joined with a groupby aggregation of pedestrians and then run needed query() calls where each row is a distinct acc_index:

veh_dict = {'1': 'bicycle', '2': 'car', '3': 'bus'}

pvt_df = (df_veh.assign(val = 1)
                .pivot_table(index = 'acc_index', 
                            columns = 'veh_type', 
                            values = 'val', 
                            aggfunc='sum')
                .set_axis([veh_dict[i] for i in list('123')], 
                          axis = 'columns',  
                          inplace = False)
                .join(df_ped.assign(pedestrians = lambda x: x['pedestrians'].astype('int'))
                            .groupby('acc_index')['pedestrians']
                            .sum()
                            .to_frame(),
                      how = 'outer'
                     )

         )

pvt_df
#            bicycle  car  bus  pedestrians
# acc_index
# 001            1.0  NaN  NaN            1
# 002            1.0  1.0  NaN            2
# 003            2.0  NaN  NaN            0
# 004            1.0  NaN  NaN            1
# 005            NaN  2.0  NaN            4
# 006            NaN  1.0  1.0            3
# 007            1.0  1.0  NaN            0
# 008            3.0  NaN  NaN            1
# 009            1.0  2.0  NaN            2

Queries

# BIKES AND CARS
pvt_df.query('(bicycle >= 1) & (car >= 1)')
#            bicycle  car  bus  pedestrians
# acc_index
# 002            1.0  1.0  0.0            2
# 007            1.0  1.0  0.0            0
# 009            1.0  2.0  0.0            2

# BIKES AND PEDESTRIANS
pvt_df.query('(bicycle >= 1) & (pedestrians >= 1)')
#            bicycle  car  bus  pedestrians
# acc_index
# 001            1.0  0.0  0.0            1
# 002            1.0  1.0  0.0            2
# 004            1.0  0.0  0.0            1
# 008            3.0  0.0  0.0            1
# 009            1.0  2.0  0.0            2

# CARS AND PEDESTRIANS
pvt_df.query('(car >= 1) & (pedestrians > 1)')
#            bicycle  car  bus  pedestrians
# acc_index
# 002            1.0  1.0  0.0            2
# 005            0.0  2.0  0.0            4
# 006            0.0  1.0  1.0            3
# 009            1.0  2.0  0.0            2

### ONLY CARS
pvt_df.query('(bicycle == 0) & (car >= 1) & (bus == 0) & (pedestrians == 0)')
# Empty DataFrame
# Columns: [bicycle, car, bus, pedestrians]
# Index: []

### ONLY BICYCLES
pvt_df.query('(bicycle >= 1) & (car == 0) & (bus == 0) & (pedestrians == 0)')
#            bicycle  car  bus  pedestrians
# acc_index
# 003            2.0  0.0  0.0            0

### ONLY PEDESTRIANS
pvt_df.query('(bicycle == 0) & (car == 0) & (bus == 0) & (pedestrians >= 1)')   
# Empty DataFrame
# Columns: [bicycle, car, bus, pedestrians]
# Index: []

Upvotes: 1

Related Questions