CODEWITHSUNDEEP
grailsgrails-orm
Olencha
Olencha

Reputation: 418

What is the best way to process a large list of Domain Objects?

What is the best way to process a large list of Domain Objects? For example, I have 'Users' and 'Book' domains, and there is a permission READ on Book object. When I'm adding a new Book, I'd like to set READ permission to all users for this Book. At first was a code:

def users = Users.findAll{ ... }
users.each { addPermission(book, it, READ) }

I'm using Spring Security Core and ACL plugin. But now, I think it is not best way to load 10000 Users Objects to memory. I gonna use the SCROLL method with maxResults(???) from Criteria. So my question is What the best way? How to determinate the best number of MaxResults?

Upvotes: 3

Views: 1036

Answers (3)

Olencha
Olencha

Reputation: 418

Thank you all. I'd like to summarize. I hope ti will be a TEMPLATE to me. 

def dc = new DetachedCriteria(Users).build{
    //some conditions of criteria 
}
def count = dc.count()

// Optional:            
// dc = dc.build{
//     projections { property('username') }
// }

def batchSize = 50 // Hibernate Doc recommends 10..50
0.step(count, batchSize){ offset-> 
    dc.list(offset:offset, max:batchSize).each{
       // doSmthWithTransaction(it)
    }
    //clear the first-level cache
    //def hiberSession = sessionFactory.getCurrentSession() 
    //hiberSession.clear()
    // or
    Users.withSession { session -> session.clear() }
}

P.S. I don't use Transaction here since I use it on the doSmthWithTransaction method

Upvotes: 0

Anuj Aneja
Anuj Aneja

Reputation: 1344

A more of Grails way to do this would be to use the batch processing. Try the example given below: EDIT : Improved answer. Now, using pagination based batch processing.

             def noOfObjectsTobeProcessedAtAtime=1000//Step or pagination size...

    List offsetMaxMapList = (0..User.count()-1).step(noOfObjectsTobeProcessedAtAtime).collect{[max:noOfObjectsTobeProcessedAtAtime,offset:it]}
    offsetMaxMapList.each{offsetMaxMap->
        addPermissionToUserInBatch(params)  

    }

    def addPermissionToUserInBatch(params){
        def batch = []
                def session
                def users = Users.createCriteria().list(params){}

                users.eachWithIndex { user, index ->
                    batch << user

                    if (batch.size() >= batchSize) {
                        User.withTransaction {
                            batch.each {User userObject ->
                             addPermission(book, userObject, READ)
                                }
                        }
                        batch.clear()

                    } else if (batch.size() < batchSize && (users.size() - index - 1) == 0) {
                        User.withTransaction {
                            batch.each {User userObject ->
                                 addPermission(book, userObject, READ)
                                }
                        }
                        batch.clear()
                    }
                    session = sessionFactory.getCurrentSession()
                    session.clear()
                }

    }

Hope that helps!!!

Upvotes: 1

Alidad
Alidad

Reputation: 5538

For something like this, I would do a bulk update. ExecuteUpdate allows you to do such a thing and its much more performant. Look at this example and customize it to your need. 

def updatedRecords = User.executeUpdate("update User set permission = 'READ' where somecriteriaOrNot ")

Upvotes: 2

Related Questions