CODEWITHSUNDEEP
pythonlotus-dominolotusscript
Ala Cris
Ala Cris

Reputation: 41

Efficient way to get data from lotus notes view

I am trying to get all data from view(Lotus Notes) with lotusscript and Python(noteslib module) and export it to csv, but problem is that this takes too much time. I have tried two ways with loop through all documents:

import noteslib
db = noteslib.Database('database','file.nsf')
view = db.GetView('My View')
doc = view.GetFirstDocument()
data = list()
while doc:
    data.append(doc.ColumnValues)
    doc = view.GetNextDocument(doc)

To get about 1000 lines of data it took me 70 seconds, but view has about 85000 lines so get all data will be too much time, because manually when I use File->Export in Lotus Notes it is about 2 minutes to export all data to csv.

And I tried second way with AllEntries, but it was even slower:

database = []
ec = view.AllEntries
ent = ec.Getfirstentry()
while ent:
    row = []
    for v in ent.Columnvalues:
        row.append(v)
    database.append(row)
    ent = ec.GetNextEntry(ent)

Everything that I found on the Internet is based on "NextDocument" or "AllEntries". Is there any way to do it faster?

Upvotes: 1

Views: 5190

Answers (3)

andora
andora

Reputation: 1366

I would suspect the performance issue is using COM/ActiveX in Python to access Notes databases. Transferring data via COM involves datatype 'marshalling', possibly at every step, and especially for 'out-of-process' method/property calls.

I don't think there is any way around this in COM. You should consider arranging a Notes 'agent' to do this for you instead (LotusScript or Java maybe). Even a basic LotusScript agent can export 000's of docs per minute. A further alternative may be to look at the Notes C-API (not an easy option and requires API calls from Python).

Upvotes: 0

Karl-Henry Martinsson
Karl-Henry Martinsson

Reputation: 2795

It is (or at least used to be) very expensive from a time standpoint to open a Notes document, like you are doing in your code. Since you are saying that you want to export the data that is being displayed in the view, you could use the NotesViewEntry class instead. It should be much faster.

Set col = view.AllEntries
Set entry = col.GetFirstEntry()
Do Until entry Is Nothing
    values = entry.ColumnValues   '*** Array of column values
    '*** Do stuff here
    Set entry = col.GetNextEntry(entry)
Loop

I wrote a blog about this back in 2013:
http://blog.texasswede.com/which-is-faster-columnvalues-or-getitemvalue/

Upvotes: 2

Tode
Tode

Reputation: 12080

Something is going on with your code "outside" the view navigation: You already chose the most performant way to navigate a view using "GetFirstDocument" and "GetNextDocument". Using the NotesViewNavigator as mentioned in the comments will be slightly better, but not significant.

You might get a little bit of performance out of your code by setting view.AutoUpdate = False to prohibit the view object to refresh when something in the backend changes. But as you only read data and not change view data that will not give you much of a performance boost.

My suggestion: Identify the REAL bottleneck of your code by commenting out single sections to find out when it starts to get slower:

First attempt:

while doc:
    doc = view.GetNextDocument(doc)

Slow?

If not then next attempt:

while doc:
    arr = doc.ColumnValues
    doc = view.GetNextDocument(doc)

Slow?

If yes: ColumnValues is your enemy... If not then next attempt: 

while doc:
    arr = doc.ColumnValues
    data.append(arr)
    doc = view.GetNextDocument(doc)

I would be very interested to get your results of where it starts to become slow.

Upvotes: 1

Related Questions