CODEWITHSUNDEEP
cachinghazelcast
andymur
andymur

Reputation: 157

Pause between hazelcast EntryProcessor process method calls

I faced behaviour of EntryProcessor that seemed a bit strange for me.

First of all what I was doing.

My task is to use EntryProcessor to fill cache B from data of another cache A.

Cache A could contain from 100K to 1M entries. So I had no choice but loop through it create B cache key from A data entry and run EntryProcessor on such key to create entry of B cache.

I found when I run such loop and build B cache from A and call EntryProcessor many times (number equals to number of entries in cache A) overall duration changes from time to time.

So I've started to write log in every step of procedure. Sometimes entry processor run with a pause between execution on several entries, see log

16:47:17.773 ce23b7a [thread-7] AppendingProcessor process process; enter 
16:47:17.773 7d9a120 [thread-7] AppendingProcessor process process; exit 

Pause between two threads (invocation on different keys) is about 10ms (which   could give us 100s on 100k keys!) 

16:47:17.782 ce23b7a [thread-0] AppendingProcessor process process; enter 
16:47:17.782 7d9a120 [thread-0] AppendingProcessor process process; exit

Sometimes we could see another behaviour, without any pause or with very small pause between execution on different entries.

14:38:42.685 ce23b7a [thread-0] AppendingProcessor process - process; enter 
14:38:42.685 7d9a120 [thread-0] AppendingProcessor process - process; exit 

14:38:42.686 ce23b7a [thread-1] AppendingProcessor process - process; enter 
14:38:42.686 7d9a120 [thread-1] AppendingProcessor process - process; exit 

14:38:42.686 ce23b7a [thread-6] AppendingProcessor process - process; enter 
14:38:42.686 7d9a120 [thread-6] AppendingProcessor process - process; exit

I thought it could be related to partitioning, so it could be related how much partitions I have and whether entryprocessor works with entries from same partition or not, but now I don't think that's the case.

Sometimes when I run the code it works on many entries with no pause, then pause (typically 10ms), then again work with no pause at all

2016-10-25 18:23:34.486 [thread-2] AppendingProcessor - process.exit; partId = 114
....
about 500 entries processed in 1 ms
...
2016-10-25 18:23:34.486 [thread-3] AppendingProcessor - process.exit; partId = 115
....
about 250 entries processed in 1 ms
...
2016-10-25 18:23:34.487 [thread-3] AppendingProcessor - process.exit; partId = 115

My question is what could be the cause of the pause between processing entries in EntryProcessor, especially if pause occurs between each two calls of process method.

Cause such 10 ms pauses don't seem right, what could be the case?

I can't provide real example of code due to NDA but I write an example of the code, see AddressMapBean#process method and appropriate AddressBookProcessor entry processor.

Any help would be appreciated!

Upvotes: 0

Views: 180

Answers (1)

enesness
enesness

Reputation: 3133

Do you read from A and put into B inside the processor? If so this has a risk of deadlock. See: https://groups.google.com/forum/#!topic/hazelcast/27_6iS4oaSY also see: https://github.com/hazelcast/hazelcast/issues/3146

Another (maybe) possible cause can be thread contention. From documentation:

NOTE: Entry Processors run via Operation Threads that are dedicated to specific partitions. Therefore, with long running Entry Processor executions, other partition operations such as map.put(key) cannot be processed. With this in mind, it is good practice to make your Entry Processor executions as quick as possible.

Upvotes: 3

Related Questions