CODEWITHSUNDEEP
solrlucene
Dean Hiller
Dean Hiller

Reputation: 20190

lucene memory growth due to sorting AND hitting all elements in index

I create an index of 2 million entries with lucene(my main program is but 30 MB RAM on startup)

I then proceed to run in a loop to search for entries sorted by time and in my loop, I make sure I hit every entry in my lucene index and immediately release it(I don't even use it). This causes the memory to grow from 30MB of RAM to 90MB of RAM. I run a jmap dump and load into eclipse after that.

the culprit turns out to be a single entry in the FieldCacheImpl$SortedDocValuesCache which has a WeakHashMap called readerCache (in the superclass of SortedDocValuesCache called Cache).

Worse yet is in production, this things grows to a size of about 40Gig (We have a 120G RAM machine). I am wondering if I can eliminate that issue in production and significantly bring RAM down with not much of a performance impact?

In FieldCacheImpl.java in the Cache inner class there is this line

final Map<Object,Map<CacheKey,Object>> readerCache = new WeakHashMap<Object,Map<CacheKey,Object>>();

notice that I am pretty sure it is just a single entry in this Map that keeps growing and growing and I guess never gets garbage collected since the index is in constant use?

Does anyone know what is going on with this? (it is a very simple test case and very reproducible).

I am using an MMapDirectory if it matters.

EDIT: oh, by the way, the key to the weak hashmap is SegmentCoreReader. A weakHashmap implies this key at some point would be garbage collected but obviously it never is garbage collected and I don't know if it gets garbage collected, will my performance tank or not nor do I know how I can ensure it gets released and garbage collected.

thanks, Dean

Upvotes: 0

Views: 246

Answers (1)

fatih
fatih

Reputation: 1395

During the sort the fieldcache fills an array with the values you sort. That makes the amount of memory you need. The cache remains to speed up the sorting the next time. The FieldCache is only released when the associated IndexReader is garbage collected.

Upvotes: 2

Related Questions