Checking Data growth in MarkLogic

Question

Does MarkLogic have a query which will show the data growth in particular database in last 1 year or let's say 6 month ? We want to know how much data growth happened in last 1 year in one of the database.

Answer 1

The documentation for afterQuery and beforeQuery states Fragment commit timestamps change not only by application transactions, but also by system transactions from the reindexer or the rebalancer. The query will also match fragments whose timestamps have been changed because of reindexing and rebalancing after the given timestamp.

So, I do not think this would truly represent the data growth.

Answer 2

Since my original answer, cts.afterQuery and cts.beforeQuery have been added which allow you accomplish the same thing without needing an additional index.

cts.estimate(cts.afterQuery(xdmp.wallclockToTimestamp(xs.date("2019-01-01"))))

If you do not have access to cts.afterQuery, see my original answer below.

---

Original Answer

In order to do this you would need to have an element/property in your data indicating a timestamp for when it was originally added to the database. For instance something like TimeOriginallyAdded. You would also need to put an element range index (or a path range index) on it.

With this, you could run a query such as :

cts.estimate(cts.jsonPropertyRangeQuery("TimeOriginallyAdded", ">=", xs.date("2019-01-01")))

Such a query would give you the total number of documents added to the database since a specified date.

Answer 3

If you have Metering enabled, you can set the retention policy to match up for how long you want to be able to look back at growth. The indexes should already exist on the Meters database to query based on time, and you should be able to gather all the data growth information needed.