how to fill date column in one dataframe with nearest dates from another dataframe

Question

I have a dataframe visit =

visit_occurrence_id  visit_start_date  person_id
    1                2016-06-01        1
    2                2019-05-01        2
    3                2016-01-22        1
    4                2017-02-14        2
    5                2018-05-11        3

and another dataframe measurement =

measurement_date    person_id   visit_occurrence_id
2017-09-04          1           Nan
2018-04-24          2           Nan
2018-05-22          2           Nan
2019-02-02          1           Nan
2019-01-28          3           Nan
2019-05-07          1           Nan
2018-12-11          3           Nan
2017-04-28          3           Nan

I want to fill the visit_occurrence_id for measurement table with visit_occurrence_id of visit table on the basis of person_id and nearest date possible.

I have written a code but its taking a lot of time.

measurement has 7*10^5 rows.

Note: visit_start_date and measurement_date are object types

my code - 

import datetime as dt

unique_person_list = measurement['person_id'].unique().tolist()

def nearest_date(row,date_list):
    date_list = [dt.datetime.strptime(date, '%Y-%m-%d').date() for date in date_list]
    row = min(date_list, key=lambda x: abs(x - row))
    return row

modified_measurement = pd.DataFrame(columns = measurement.columns)

for person in unique_person_list:
    near_visit_dates =  visit[visit['person_id']==person]['visit_start_date'].tolist()
    if near_visit_dates:
        near_visit_dates = list(filter(None, near_visit_dates))
        near_visit_dates = [i.strftime('%Y-%m-%d') for i in near_visit_dates]
        store_dates = measurement.loc[measurement['person_id']== person]['measurement_date']
        store_dates= store_dates.apply(nearest_date, args=(near_visit_dates,))
        modified_measurement = modified_measurement.append(store_dates)

My code's execution time is quite high. Can you help me in either reducing the time complexity or with another solution.

edit - adding dataframe constructors.

import numpy as np

measurement = {'measurement_date':['2017-09-04', '2018-04-24', '2018-05-22', '2019-02-02', 
                                   '2019-01-28', '2019-05-07', '2018-12-11','2017-04-28'],
        'person_id':[1, 2, 2, 1, 3, 1, 3, 3],'visit_occurrence_id':[np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan]}

visit = {'visit_occurrence_id':[1, 2, 3, 4, 5], 
         'visit_start_date':['2016-06-01', '2019-05-01', '2016-01-22', '2017-02-14', '2018-05-11'],
         'person_id':[1, 2, 1, 2, 3]}

# Create DataFrame
measurement = pd.DataFrame(measurement)
visit = pd.DataFrame(visit)

Answer 1

Here's what I've come up with:

# Get all visit start dates
df = measurement.drop('visit_occurrence_id', axis=1).merge(visit, on='person_id')
df['date_difference'] = abs(df.measurement_date - df.visit_start_date)
# Find the smallest visit start date for each person_id - measurement_date pair
df['smallest_difference'] = df.groupby(['person_id', 'measurement_date'])['date_difference'].transform(min)
df = df[df.date_difference == df.smallest_difference]
df = df[['measurement_date', 'person_id', 'visit_occurrence_id']]
# Fill in visit_occurrence_id from original dataframe
measurement.drop("visit_occurrence_id", axis=1).merge(
    df, on=["measurement_date", "person_id"]
)

This produces:

|    | measurement_date   |   person_id |   visit_occurrence_id |
|---:|:-------------------|------------:|----------------------:|
|  0 | 2017-09-04         |           1 |                     1 |
|  1 | 2018-04-24         |           2 |                     2 |
|  2 | 2018-05-22         |           2 |                     2 |
|  3 | 2019-02-02         |           1 |                     1 |
|  4 | 2019-01-28         |           3 |                     5 |
|  5 | 2019-05-07         |           1 |                     1 |
|  6 | 2018-12-11         |           3 |                     5 |
|  7 | 2017-04-28         |           3 |                     5 |

I believe there's probably a cleaner way of writing this using sklearn: https://scikit-learn.org/stable/modules/neighbors.html

Answer 2

You can do the following:

df=pd.merge(measurement[["person_id", "measurement_date"]], visit, on="person_id", how="inner")

df["dt_diff"]=df[["visit_start_date", "measurement_date"]].apply(lambda x: abs(datetime.datetime.strptime(x["visit_start_date"], '%Y-%m-%d').date() - datetime.datetime.strptime(x["measurement_date"], '%Y-%m-%d').date()), axis=1)

df=pd.merge(df, df.groupby(["person_id", "measurement_date"])["dt_diff"].min(), on=["person_id", "dt_diff", "measurement_date"], how="inner")

res=pd.merge(measurement, df, on=["measurement_date", "person_id"], suffixes=["", "_2"])[["measurement_date", "person_id", "visit_occurrence_id_2"]]

Output:

  measurement_date  person_id  visit_occurrence_id_2
0       2017-09-04          1                      1
1       2018-04-24          2                      2
2       2018-05-22          2                      2
3       2019-02-02          1                      1
4       2019-01-28          3                      5
5       2019-05-07          1                      1
6       2018-12-11          3                      5
7       2017-04-28          3                      5