Ontotext Parameter defaut.min.distinct.threshold not honored

Question

Ontotext database throws NotEnoughMemoryForDistinctGroupBy: Insufficient free Heap Memory ... , threshold:250Mb although parameter defaut.min.distinct.threshold=1074000000 set.

It seems that setting the parameter is not taken into account by the Ontotext engine und thus we are still getting the error.

We are using Ontotext 8.10.1-Free in Docker. Running the container we provide as much memory to the container as is set for the Xmx parameter

We tried different memory settings Xms, Xmx and the aforementioned threshold parameter. But in the logs it states that the threshold's default value of 250m is being used.

This is how our process info looks at runtime:

>  ps auxw|grep java
root         1 85.4 47.1 5729308 964184 ?      Ssl  09:24   0:46 /docker-java-home/bin/java -XX:MaxDirectMemorySize=128G -Xms2g -Xmx2g -Ddefaut.min.distinct.threshold=1074000000  
-Djava.net.preferIPv4Stack=true -XX:+UseParallelGC -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/graphdb/dist/heapdump.hprof -XX:OnOutOfMemoryError=kill -9 %p
-Djava.awt.headless=true -Dfile.encoding=UTF-8 -Dgraphdb.foreground=yes -Dgraphdb.dist=/opt/graphdb/dist -cp /opt/graphdb/dist/lib/* -Dgraphdb.home=/opt/graphdb/home com.ontotext.graphdb.server.GraphDBWorkbench

However, when we look into the logs (main-... .log) we see statements like these (mind the line where I added 2 stars at the beginning):

[INFO ] 2019-08-08 09:33:22,793 [repositories/kg-test-graph-import | c.o.c.AbstractParameter] Configured parameter 'FSIPC' to default value '2'
[INFO ] 2019-08-08 09:33:23,138 [repositories/kg-test-graph-import | c.o.g.s.StatementsController] POST SPARQL update request to repository
[INFO ] 2019-08-08 09:33:23,244 [repositories/kg-test-graph-import | c.o.g.s.StatementsController] POST SPARQL update request to repository
[INFO ] 2019-08-08 09:33:23,299 [repositories/kg-test-graph-import | c.o.f.s.RepositoryController] GET query -1149340244
[INFO ] 2019-08-08 09:33:23,334 [repositories/kg-test-graph-import | c.o.c.AbstractParameter] Configured parameter 'order.construct' to default value 'false'
**[INFO ] 2019-08-08 09:33:23,345 [repositories/kg-test-graph-import | c.o.c.AbstractParameter] Configured parameter 'default.min.distinct.threshold' to default value '250.0M'
[INFO ] 2019-08-08 09:33:23,357 [repositories/kg-test-graph-import | c.o.g.s.GraphQueryResultView] Request for query -1149340244 is finished
[INFO ] 2019-08-08 09:33:23,367 [repositories/kg-test-graph-import | c.o.g.s.StatementsController] POST SPARQL update request to repository
[INFO ] 2019-08-08 09:33:23,415 [repositories/kg-test-graph-import | c.o.g.s.StatementsController] POST SPARQL update request to repository
[INFO ] 2019-08-08 09:33:23,487 [repositories/kg-test-graph-import | c.o.g.s.StatementsController] POST SPARQL update request to repository
[INFO ] 2019-08-08 09:33:23,704 [repositories/kg-test-graph-import | c.o.f.s.RepositoryController] GET query 1759975767

Moreover the ontotext logs show the exceptions that can also be witnessed in the client running our tests and Sparql request against the DB:

[WARN ] 2019-08-08 09:41:10,442 [repositories/kg-test-graph-import | c.o.t.u.DistinctMemoryMonitor] Insufficient free Heap Memory 238Mb for group by and distinct, threshold 250Mb,  reached 0Mb
[ERROR] 2019-08-08 09:41:10,469 [repositories/kg-test-graph-import | c.o.t.SailConnectionImpl] Memory limit for GroupBy/Distinct reached
com.ontotext.trree.util.NotEnoughMemoryForDistinctGroupBy: Insufficient free Heap Memory 238Mb for group by and distinct, threshold:250Mb, reached 0Mb
        at com.ontotext.trree.util.DistinctMemoryMonitor.updateGroupByMemory(DistinctMemoryMonitor.java:73)
        at com.ontotext.trree.util.LongKeyBase.(LongKeyBase.java:38)
        at com.ontotext.trree.util.LongKeySet.(LongKeySet.java:26)
        at com.ontotext.trree.query.OwlimDistinctIteration.(OwlimDistinctIteration.java:20)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:1086)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:639)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:228)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.TupleFunctionEvaluationStrategy.evaluate(TupleFunctionEvaluationStrategy.java:65)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:135)
        at org.eclipse.rdf4j.query.algebra.evaluation.iterator.JoinIterator.(JoinIterator.java:42)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:824)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:237)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:796)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:230)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.TupleFunctionEvaluationStrategy.evaluate(TupleFunctionEvaluationStrategy.java:65)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:135)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:650)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:228)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.TupleFunctionEvaluationStrategy.evaluate(TupleFunctionEvaluationStrategy.java:65)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:135)
        at com.ontotext.trree.SailConnectionImpl.evaluateWithRepoConnectionInternal(SailConnectionImpl.java:1786)
        at com.ontotext.trree.SailConnectionImpl.evaluateWithRepoConnection(SailConnectionImpl.java:1679)
        at com.ontotext.trree.SailConnectionImpl.evaluateInternal(SailConnectionImpl.java:249)
        at com.ontotext.trree.SailConnectionImpl.evaluate(SailConnectionImpl.java:2316)
        at org.eclipse.rdf4j.repository.sail.helpers.SailUpdateExecutor.evaluateWhereClause(SailUpdateExecutor.java:487)
        at org.eclipse.rdf4j.repository.sail.helpers.SailUpdateExecutor.executeModify(SailUpdateExecutor.java:450)
        at org.eclipse.rdf4j.repository.sail.helpers.SailUpdateExecutor.executeUpdate(SailUpdateExecutor.java:121)
        at org.eclipse.rdf4j.repository.sail.SailUpdate.execute(SailUpdate.java:64)
        at com.ontotext.trree.monitorRepository.MonitorRepositoryConnection$1.execute(MonitorRepositoryConnection.java:382)
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.getSparqlUpdateResult(StatementsController.java:254)
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.handleRequestInternal(StatementsController.java:114)
        at com.ontotext.graphdb.sesame.StatementsController.handleRequestInternal(StatementsController.java:41)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.ontotext.forest.core.request.RequestFilter.doFilterInternal(RequestFilter.java:45)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.ontotext.forest.security.AdminDelegatingFilterProxy.doFilter(AdminDelegatingFilterProxy.java:38)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
[ERROR] 2019-08-08 09:41:10,549 [repositories/kg-test-graph-import | c.o.f.s.GraphDBProtocolExceptionResolver] Error while handling request (500)
org.eclipse.rdf4j.http.server.ServerHTTPException: org.eclipse.rdf4j.query.UpdateExecutionException: org.eclipse.rdf4j.sail.SailException: com.ontotext.trree.util.NotEnoughMemoryForDistinctGroupBy: Insufficient free Heap Memory 238Mb for group by and distinct, threshold:250Mb, reached 0Mb
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.getSparqlUpdateResult(StatementsController.java:266)
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.handleRequestInternal(StatementsController.java:114)
        at com.ontotext.graphdb.sesame.StatementsController.handleRequestInternal(StatementsController.java:41)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.ontotext.forest.core.request.RequestFilter.doFilterInternal(RequestFilter.java:45)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.ontotext.forest.security.AdminDelegatingFilterProxy.doFilter(AdminDelegatingFilterProxy.java:38)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.rdf4j.query.UpdateExecutionException: org.eclipse.rdf4j.sail.SailException: com.ontotext.trree.util.NotEnoughMemoryForDistinctGroupBy: Insufficient free Heap Memory 238Mb for group by and distinct, threshold:250Mb, reached 0Mb
        at org.eclipse.rdf4j.repository.sail.SailUpdate.execute(SailUpdate.java:69)
        at com.ontotext.trree.monitorRepository.MonitorRepositoryConnection$1.execute(MonitorRepositoryConnection.java:382)
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.getSparqlUpdateResult(StatementsController.java:254)
        ... 45 common frames omitted
Caused by: org.eclipse.rdf4j.sail.SailException: com.ontotext.trree.util.NotEnoughMemoryForDistinctGroupBy: Insufficient free Heap Memory 238Mb for group by and distinct, threshold:250Mb, reached 0Mb
        at com.ontotext.trree.SailConnectionImpl.evaluateWithRepoConnectionInternal(SailConnectionImpl.java:1815)
        at com.ontotext.trree.SailConnectionImpl.evaluateWithRepoConnection(SailConnectionImpl.java:1679)
        at com.ontotext.trree.SailConnectionImpl.evaluateInternal(SailConnectionImpl.java:249)
        at com.ontotext.trree.SailConnectionImpl.evaluate(SailConnectionImpl.java:2316)
        at org.eclipse.rdf4j.repository.sail.helpers.SailUpdateExecutor.evaluateWhereClause(SailUpdateExecutor.java:487)
        at org.eclipse.rdf4j.repository.sail.helpers.SailUpdateExecutor.executeModify(SailUpdateExecutor.java:450)
        at org.eclipse.rdf4j.repository.sail.helpers.SailUpdateExecutor.executeUpdate(SailUpdateExecutor.java:121)
        at org.eclipse.rdf4j.repository.sail.SailUpdate.execute(SailUpdate.java:64)
        ... 47 common frames omitted
Caused by: com.ontotext.trree.util.NotEnoughMemoryForDistinctGroupBy: Insufficient free Heap Memory 238Mb for group by and distinct, threshold:250Mb, reached 0Mb
        at com.ontotext.trree.util.DistinctMemoryMonitor.updateGroupByMemory(DistinctMemoryMonitor.java:73)
        at com.ontotext.trree.util.LongKeyBase.(LongKeyBase.java:38)
        at com.ontotext.trree.util.LongKeySet.(LongKeySet.java:26)
        at com.ontotext.trree.query.OwlimDistinctIteration.(OwlimDistinctIteration.java:20)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:1086)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:639)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:228)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.TupleFunctionEvaluationStrategy.evaluate(TupleFunctionEvaluationStrategy.java:65)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:135)
        at org.eclipse.rdf4j.query.algebra.evaluation.iterator.JoinIterator.(JoinIterator.java:42)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:824)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:237)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:796)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:230)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.TupleFunctionEvaluationStrategy.evaluate(TupleFunctionEvaluationStrategy.java:65)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:135)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:650)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.StrictEvaluationStrategy.evaluate(StrictEvaluationStrategy.java:228)
        at org.eclipse.rdf4j.query.algebra.evaluation.impl.TupleFunctionEvaluationStrategy.evaluate(TupleFunctionEvaluationStrategy.java:65)
        at com.ontotext.trree.query.OwlimEvaluationStrategyImpl.evaluate(OwlimEvaluationStrategyImpl.java:135)
        at com.ontotext.trree.SailConnectionImpl.evaluateWithRepoConnectionInternal(SailConnectionImpl.java:1786)
        ... 54 common frames omitted
[INFO ] 2019-08-08 09:41:10,666 [repositories/kg-test-graph-import | c.o.g.s.StatementsController] POST SPARQL update request to repository

What are our options to mitigate the situation? Currently I am not able to proceed with our software to get it up and running again due to the failing tests. Backround here is that we are evaluating to switch to Ontotext from another triple store.

Lots of thanks and

best regards Tom

Ontotext Parameter defaut.min.distinct.threshold not honored

Answers (1)

Related Questions