Reputation: 2275
We changed the Coherence cache from "replicated" to "near-distributed-persistent", but now the servers get overloaded.
This is the current configuration:
<?xml version='1.0'?>
<cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config">
<caching-scheme-mapping>
<cache-mapping>
<cache-name>app-*</cache-name>
<scheme-name>app-refdata-near</scheme-name>
</cache-mapping>
<cache-mapping>
<cache-name>configConsultationCache</cache-name>
<scheme-name>config</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<near-scheme>
<scheme-name>app-refdata-near</scheme-name>
<front-scheme>
<local-scheme>
<scheme-ref>app-refdata-local</scheme-ref>
</local-scheme>
</front-scheme>
<back-scheme>
<distributed-scheme>
<scheme-ref>partitioned</scheme-ref>
</distributed-scheme>
</back-scheme>
</near-scheme>
<local-scheme>
<scheme-name>app-refdata-local</scheme-name>
</local-scheme>
<distributed-scheme>
<scheme-name>partitioned</scheme-name>
<backup-count>2</backup-count>
<backing-map-scheme>
<local-scheme/>
</backing-map-scheme>
<persistence>
<environment>default-active</environment>
</persistence>
<autostart>true</autostart>
</distributed-scheme>
<local-scheme>
<scheme-name>config</scheme-name>
<expiry-delay>1h</expiry-delay>
</local-scheme>
</caching-schemes>
</cache-config>
Which, by the way, is duplicated: is present in 2 jar, but under the same deployable ear.
In the Oracle documentation, it is specified that for active-persistent mode, an additional allocation of 20-35% more memory is recommended, which has already been done.
Each node has approximately 50g of physical memory available, and we have set the heap size between -Xmx=32g and -Xms=32g, but even with that, the memory usage reaches more than 90% of the total available.
On a nightly basis, the application refreshes the content of the cache.
It has been noticed that the overload becomes more pronounced when the application is used intensively (it's a high-throughput type of application), as a result, the cache is used (read) further.
By checking the MBeans of Coherence in the Enterprise Manager, we see multiple entries for the same cache name. For instance, 3 of them are NearCache types and the other 3 are LocalCache.
We have analyzed a heap dump, but there is nothing unusual. The heap dump weighs just 9g, not 32g. The biggest objects are, as expected, from Coherence.
Any advice? Does the coherence configuration look okay? Does the refresh of the complete cache conflict with the new configuration?
Upvotes: 0
Views: 144