CODEWITHSUNDEEP
groovy
user3679267
user3679267

Reputation: 1

Groovy GPars Script

GParsPool.withPool(numberPool) {

            connection.withBatch(10000) { stmt ->
                inputFile.eachParallel { data ->

                    //GParsPool.withPool() {
                        stmt.addBatch("DELETE FROM user WHERE number = ${data.toLong()} ")
                        println "IN"
                    //}
                }

                println "OUT"
                Long startTimee = System.currentTimeMillis()

                stmt.executeBatch()
                println "deleted Batch"

                Long endTime = System.currentTimeMillis()
                println "Time taken for each batch: " + ((endTime - startTimee) / 1000)

            }
        }

The Above code is used to delete data from database. I First get the data from file and then match the each file data with database data and perform the delete query. But I have 5533179 records its take to much time. Even I have used the gpars but i get the same performance issue that is given by without using gpars. I have set the numberPool=5 but same issue. Even I increase the numberPool again same issue

Upvotes: 0

Views: 359

Answers (1)

injecteer
injecteer

Reputation: 20699

why don't you use the SQL in operator? thus you can process the data much faster.

UPDATE:

from the top of my head:

GParsPool.withPool(numberPool) {
    Map buffPerThread = [:].withDefaults{ [] }

    inputFile.eachParallel { data ->
      def buff = buffPerThread[ Thread.currentThread().id ]
      buff << data.toLong()
      if( 1000 == buff.size() ){
        sql.execute 'DELETE FROM user WHERE number in (?)', [ buff ]
        buff.clear()
      }
    }
 }

I wouldn't use conn.withBatch here, as the in statement give the desired batching already

Upvotes: 1

Related Questions