CODEWITHSUNDEEP
elasticsearchindexingdocument
Damitha Dayananda
Damitha Dayananda

Reputation: 493

Elasticsearch document count doesn't reflect higher indexing rate

When we monitor our elasticsearch cluster health through kibana, For a particular index we see very higher indexing rate. But it seems document count doesn't increase proportionally. How to tackle these two.

document count

indexing rate

sample document

{ "_index": "finance_report_fgl_reporting_log", "_type": "fgl_reporting_logs", "_id": "1907688_POINTS_ACCOUNT_DEBIT", "_score": 9.445704, "_source": { "reportingLogId": { "journalId": 1907688, "postingAccountId": "POINTS_ACCOUNT", "postingAccountingEntry": "DEBIT" }, "journalId": 1907688, "journalEventId": "trip_completed", "journalEventLogId": "15db1f2b-b9d0-4edd-96f0-c4e4f8e68150", "journalAccountingRuleId": "trip_completed_points_payment_rule", "journalReferenceId": "174558200", "journalGrossAmount": 154.11, "postingJournalId": 1907688, "postingAccountingRuleId": "trip_completed_points_payment_rule", "postingReferenceId": "174558200", "postingAccountId": "POINTS_ACCOUNT", "postingAccountingPeriod": "2019_08", "postingAccountingEntry": "DEBIT", "postingCurrencyTypeId": "POINTS", "postingAmount": 154.11, "accountId": "POINTS_ACCOUNT", "accountStakeholderId": "OPERATOR", "accountCurrencyTypeId": "POINTS", "accountTypeId": "CONTROLLER", "accountingRuleId": "trip_completed_points_payment_rule", "accountingRuleDescription": "Points payment", "eventId": "trip_completed", "eventReferenceParam": "body.trip.id", "createdDate": "2019-08-29T10:03:32.000+0530", "modifiedDate": "2019-08-29T10:03:32.000+0530", "createdBy": "ENGINE", "modifiedBy": "ENGINE", "version": "3.12.6", "createYear": 2019, "routingKey": "_2019" } },

Upvotes: 0

Views: 581

Answers (2)

ugosan
ugosan

Reputation: 1586

You might get some info when performing GET _cat/indices?v check out the "docs.deleted" column, as an update operation is merely a "create new+delete older" operation.

Upvotes: 0

Val
Val

Reputation: 217544

The reason why this usually happens is because your indexing operations do not create new documents but update existing ones. Mainly because you're sending updates to an ID that already exists.

Every few hours, a new batch of documents is created (according to the jumps in the graphs) because you're creating a new set of IDs.

Make sure to verify how you're creating your IDs as the solution is hidden in there somewhere.

Upvotes: 0

Related Questions