CODEWITHSUNDEEP
pythonpyodbc
John
John

Reputation: 1821

Creation pyodbc cursor to multiple databases

I am processing events stored in MS-Access databases using pyodbc. Each month is a seperate file / database and I would like to process events from multiple months.

Is it possible to create a cursor to a view containing multiple months i.e. database connections?

Edit 1: And without having to write a new database? (Something like UNION VIEW maybe?)

Upvotes: 0

Views: 3906

Answers (1)

bbayles
bbayles

Reputation: 4527

You'll need to make multiple connections and cursors, but you should be able to process the data.

Let's say the files are stored as month_1.mdb, month_2.mdb, etc. in C:\access.

# Set up each connection, must have a way to access each file's name
connect_string = "Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\\access\\month_{}.mdb;"
# Assuming that you'll get the same data from each database
sql = "SELECT column_1, column_2 FROM table"
# Connect to each file
connections = [pyodbc.connect(connect_string.format(n)) for n in range(1, 12 + 1)]
# Create a cursor for each file
cursors = [conn.cursor() for conn in connections]
# Query each file and save the data
data = []
for cur in cursors:
  cur.execute(sql)
  data.extend(cur.fetchall())

OK, so now you have all the data. You can create an in-memory database with the sqlite3 module and then do queries against it.

import sqlite3
# Create your temporary database
connection = sqlite3.connect(":memory:")
cursor = connection.cursor()
# Set up a place to hold the data fetched previously
_ = cur.execute("CREATE TABLE t(x INTEGER, y INTEGER)")
# Dump all the data into the database
for column_1, column_2 in data:
  _ = cursor.execute("INSERT INTO t VALUES (?, ?)", [column_1, column_2])
# Now you can run queries against the new view of your data
sql = "SELECT t.column_1, t.count(*) FROM t GROUP BY t.column_1"

Upvotes: 4

Related Questions