Limit MarkLogic memory consumption in docker container

Question

The project in which I am working develops a Java service that uses MarkLogic 9 in the backend. We are running a Jenkins build server that executes (amongst others) several tests in MarkLogic written in XQuery. For those tests MarkLogic is running in a docker container on the Jenkins host (which is running Ubuntu Linux). The Jenkins host has 12 GB of RAM and 8 GB of swap configured. Recently I have noticed that the MarkLogic instance running in the container uses a huge amount of RAM (up to 10 GB). As there are often other build jobs running in parallel, the Jenkins starts to swap, sometimes even eating up all swap so that MarkLogic reports it cannot get more memory. Obviously, this situation leads to failed builds quite often.

To analyse this further I made some tests on my PC running Docker for Windows and found out that the MarkLogic tests can be run successfully with 5-6 GB RAM. The MarkLogic logs show that it sees all the host memory and wants to use everything. But as we have other build processes running on that host this behaviour is not desirable.

My question: is there any possibility to tell the MarkLogic to not use so much memory? We are preparing the docker image during the build, so we could modify some configuration, but it has to be scripted somehow.

Answer 1

The issue of the container not detecting memory limit correctly has been identified, and should be addressed in a forthcoming release.

In the meantime, you might be able to mitigate the issue by:

• changing the group cache sizing from automatic to manual and setting cache sizes appropriate for the allocated resources. There area variety of ways to set these configs, whether deploying and settings configs from ml-gradle project, making your own Manage API REST calls, or programmatically:

  • admin:group-set-cache-sizing
  • admin:group-set-compressed-tree-cache-partitions
  • admin:group-set-compressed-tree-cache-size
  • admin:group-set-expanded-tree-cache-partitions
  • admin:group-set-expanded-tree-cache-size
  • admin:group-set-list-cache-partitions
  • admin:group-set-list-cache-size

• reducing the in-memory-limit

  in memory limit specifies the maximum number of fragments in an in-memory stand. An in-memory stand contains the latest version of any new or changed fragments. Periodically, in-memory stands are written to disk as a new stand in the forest. Also, if a stand accumulates a number of fragments beyond this limit, it is automatically saved to disk by a background thread.