CODEWITHSUNDEEP
pythonpandasms-accesspyodbc
Mtra
Mtra

Reputation: 201

Insert or update rows in MS Access database in Python

I've got an MS Access table (SearchAdsAccountLevel) which needs to be updated frequently from a python script. I've set up the pyodbc connection and now I would like to UPDATE/INSERT rows from my pandas df to the MS Access table based on whether the Date_ AND CampaignId fields match with the df data.

Looking at previous examples I've built the UPDATE statement which uses iterrows to iterate through all rows within df and execute the SQL code as per below:

    connection_string = (
            r"Driver={Microsoft Access Driver (*.mdb, *.accdb)};"
            r"c:\AccessDatabases\Database2.accdb;"
    )
    cnxn = pyodbc.connect(connection_string, autocommit=True)
    crsr = cnxn.cursor()

    for index, row in df.iterrows():
            crsr.execute("UPDATE SearchAdsAccountLevel SET [OrgId]=?, [CampaignName]=?, [CampaignStatus]=?, [Storefront]=?, [AppName]=?, [AppId]=?, [TotalBudgetAmount]=?, [TotalBudgetCurrency]=?, [DailyBudgetAmount]=?, [DailyBudgetCurrency]=?, [Impressions]=?, [Taps]=?, [Conversions]=?, [ConversionsNewDownloads]=?, [ConversionsRedownloads]=?, [Ttr]=?, [LocalSpendAmount]=?, [LocalSpendCurrency]=?, [ConversionRate]=?, [Week_]=?, [Month_]=?, [Year_]=?, [Quarter]=?, [FinancialYear]=?, [RowUpdatedTime]=? WHERE [Date_]=? AND [CampaignId]=?",
                        row['OrgId'],
                        row['CampaignName'],
                        row['CampaignStatus'],
                        row['Storefront'],
                        row['AppName'],
                        row['AppId'],
                        row['TotalBudgetAmount'],
                        row['TotalBudgetCurrency'],
                        row['DailyBudgetAmount'],
                        row['DailyBudgetCurrency'],
                        row['Impressions'],
                        row['Taps'],
                        row['Conversions'],
                        row['ConversionsNewDownloads'],
                        row['ConversionsRedownloads'],
                        row['Ttr'],
                        row['LocalSpendAmount'],
                        row['LocalSpendCurrency'],
                        row['ConversionRate'],
                        row['Week_'],
                        row['Month_'],
                        row['Year_'],
                        row['Quarter'],
                        row['FinancialYear'],
                        row['RowUpdatedTime'],
                        row['Date_'],
                        row['CampaignId'])
crsr.commit()

I would like to iterate through each row within my df (around 3000) and if the ['Date_'] AND ['CampaignId'] match I UPDATE all other fields. Otherwise I want to INSERT the whole df row in my Access Table (create new row). What's the most efficient and effective way to achieve this?

Upvotes: 2

Views: 6102

Answers (1)

Parfait
Parfait

Reputation: 107567

Consider DataFrame.values and pass list into an executemany call, making sure to order columns accordingly for the UPDATE query:

cols = ['OrgId', 'CampaignName', 'CampaignStatus', 'Storefront',
        'AppName', 'AppId', 'TotalBudgetAmount', 'TotalBudgetCurrency',
        'DailyBudgetAmount', 'DailyBudgetCurrency', 'Impressions',
        'Taps', 'Conversions', 'ConversionsNewDownloads', 'ConversionsRedownloads',
        'Ttr', 'LocalSpendAmount', 'LocalSpendCurrency', 'ConversionRate',
        'Week_', 'Month_', 'Year_', 'Quarter', 'FinancialYear',
        'RowUpdatedTime', 'Date_', 'CampaignId']

sql = '''UPDATE SearchAdsAccountLevel 
            SET [OrgId]=?, [CampaignName]=?, [CampaignStatus]=?, [Storefront]=?, 
                [AppName]=?, [AppId]=?, [TotalBudgetAmount]=?, 
                [TotalBudgetCurrency]=?, [DailyBudgetAmount]=?, 
                [DailyBudgetCurrency]=?, [Impressions]=?, [Taps]=?, [Conversions]=?, 
                [ConversionsNewDownloads]=?, [ConversionsRedownloads]=?, [Ttr]=?, 
                [LocalSpendAmount]=?, [LocalSpendCurrency]=?, [ConversionRate]=?,
                [Week_]=?, [Month_]=?, [Year_]=?, [Quarter]=?, [FinancialYear]=?, 
                [RowUpdatedTime]=? 
          WHERE [Date_]=? AND [CampaignId]=?'''

crsr.executemany(sql, df[cols].values.tolist())   
cnxn.commit()

For the insert, use a temp, staging table with exact structure as final table which you can create with make-table query: SELECT TOP 1 * INTO temp FROM final. This temp table will be regularly cleaned out and inserted with all data frame rows. The final query migrates only new rows from temp into final with NOT EXISTS, NOT IN, or LEFT JOIN/NULL. You can run this query anytime and never worry about duplicates per Date_ and CampaignId columns.

# CLEAN OUT TEMP
sql = '''DELETE FROM SearchAdsAccountLevel_Temp'''
crsr.executemany(sql)   
cnxn.commit()

# APPEND TO TEMP
sql = '''INSERT INTO SearchAdsAccountLevel_Temp (OrgId, CampaignName, CampaignStatus, Storefront,
                                AppName, AppId, TotalBudgetAmount, TotalBudgetCurrency,
                                DailyBudgetAmount, DailyBudgetCurrency, Impressions,
                                Taps, Conversions, ConversionsNewDownloads, ConversionsRedownloads,
                                Ttr, LocalSpendAmount, LocalSpendCurrency, ConversionRate,
                                Week_, Month_, Year_, Quarter, FinancialYear,
                                RowUpdatedTime, Date_, CampaignId)    
         VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 
                 ?, ?, ?, ?, ?, ?, ?, ?, ?, 
                 ?, ?, ?, ?, ?, ?, ?, ?, ?);'''

crsr.executemany(sql, df[cols].values.tolist())   
cnxn.commit()

# MIGRATE TO FINAL
sql = '''INSERT INTO SearchAdsAccountLevel 
         SELECT t.* 
         FROM SearchAdsAccountLevel_Temp t
         LEFT JOIN SearchAdsAccountLevel f
            ON t.Date_ = f.Date_ AND t.CampaignId = f.CampaignId
         WHERE f.OrgId IS NULL'''
crsr.executemany(sql)   
cnxn.commit()

Upvotes: 3

Related Questions