CODEWITHSUNDEEP
ms-accessimportexcelvba
Bruder
Bruder

Reputation: 573

I need to dynamically (and quickly) import part of the data in an Access table (a large one) into Excel

I'm trying to create a dynamic report in Excel. I have lots of sales data and I want the user to be able to slice and dice it according to his needs.

Normally I would use two sheets: one hidden, containing the raw data, and one visible, containing all the buttons and form controls so that the user can dinamically select and visualize only a small subset of the original data at the time.

The problem is that this time I need to handle 6.000.000+ rows of data (and counting). Storing it all in an excel sheet is not an option. Besides, the data is already in the form of an Access table.

I tried accessing it dinamically via a query that "filters out" the un-needed information based on what the user selects in the form control on the Excel sheet. For some reason, this is very slow. It takes 4-5 minutes to pull out as little as 10 rows of data.

There has to be a quicker way to do this! I need this whole process to feel "instantaneous".

Any thoughts?

Edit: Ok, so the problem seems to be related to the fact that my access table is actually a linked table pointing to a *.txt file. This slows the import down a lot.

I tried both of the suggested solutions.

iDevlop's idea works quite fast (200k rows imported in 10-15 secs), but it has the downside of me having to update the table every time. I'll post another question, like he suggested, to see how and if the process can be automated.

Remou's script works perfectly too now (I had a hard time getting it right but he was really open and helpful so know I got it) and, although slower, it has the advantage of not requiring any database mantainance.

There's a few more things I need to get straight before choosing which approach to use. For now, all I want to say is thank you guys for you help! I could have never made it without you!!!

Upvotes: 2

Views: 1735

Answers (2)

Fionnuala
Fionnuala

Reputation: 91316

Don't bother going through Access if you have a text file. This may hold you until you can get a better system in place.

Dim cn As Object
Dim rs As Object
Dim strFile As String
Dim strCon As String
Dim strSQL As String
Dim s As String
Dim i As Integer, j As Integer

strFile = "z:\docs\"

''Note that if HDR=No, F1,F2 etc are used for column names,
''if HDR=Yes, the names in the first row of the range
''can be used.
''
''Connection strings : http://www.connectionstrings.com/excel

strCon = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & strFile _
    & ";Extended Properties=""Text;HDR=Yes;IMEX=1"";"

''Late binding, so no reference is needed

Set cn = CreateObject("ADODB.Connection")
Set rs = CreateObject("ADODB.Recordset")

cn.Open strCon

strSQL = "SELECT * " _
       & "FROM [test.txt] a " _
       & "WHERE a.FirstName ='Bernard'"

rs.Open strSQL, cn, 3, 3


''Pick a suitable empty worksheet for the results

Worksheets("Sheet3").Cells(2, 1).CopyFromRecordset rs

''Tidy up
rs.Close
Set rs = Nothing
cn.Close
Set cn = Nothing

If your file is tab delimited, you can use a schema.ini (http://msdn.microsoft.com/en-us/library/ms709353(VS.85).aspx). It must be in the same directory as you text file and need only contain two lines:

[Ordini BO new.txt]
Format=TabDelimited

Your connection string should read:

strCon = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & strFile _
& ";Extended Properties=""Text;HDR=No;IMEX=1;FMT=Delimited"";"

Upvotes: 2

iDevlop
iDevlop

Reputation: 25252

As says Remou, check xour indexes, but also make sure your criteria are entered in a way that allows indexes to be used !
e.g : WHERE Format(myDate,"yyyy-mm") = "2011-09" does not allow the date index optimisation,
while WHERE myDate BETWEEN #09/01/2011# AND #09/30/2011# does allow index optimisation.

Edit:
If you have some kind of unique identifier in your text file and you translate that into a PK in your table design, you can then import the whole thing on a regular basis, and the duplicates will be discarded by the PK.
The import could be automated, even with a .vbs, you don't need Access to do it. Make that another question if you're stuck. You could also ask the IT guys to delete the older records every month or so.

Upvotes: 2

Related Questions