Reputation: 648
I have mongodb database for storing all the data, from few collections I'm syncing MongoDB database data to AWS Open Search via Monstache Plugin. For normal case working fine but when some of my mongodb data updated so frequently like Status field in that case that status is not updated into the AWS Open Search but in MongoDB is updated correctly.
Here is monstache logs
{
"doc": {
"CdxActivityId": 417462,
"CdxClusterId": "services.*.com",
"CdxCreatedAt": "2025-03-03T06:15:28.000Z",
"InternalGroupId": "*",
**"__v": 0,**
"contentType": "application/octet-stream",
"createdAt": "2025-03-03T05:15:28.954Z",
"linkedActivities": [],
"location": "IE",
"oplog_date": "2025/03/03 05:15:28",
"oplog_ts": {
"T": 1740978928,
"I": 7
},
"restartEvents": [],
"restartHistory": [],
"status": "No statuses"
},
"doc_as_upsert": true
}
In mongodb we have __v field in which we are maintaining the version and in first case __v=0 I got no status which updated correctly in Open search via monstache syncing.
But very frequently when I got next status like Success then that status not present in open search
{
"doc": {
"CdxActivityId": 417462,
"CdxClusterId": "services.*.com",
"CdxCreatedAt": "2025-03-03T06:15:28.000Z",
"InternalGroupId": "*",
"__v": 2,
"contentType": "application/octet-stream",
"createdAt": "2025-03-03T05:15:28.954Z",
"linkedActivities": [],
"location": "IE",
"oplog_date": "2025/03/03 05:15:29",
"oplog_ts": {
"T": 1740978929,
"I": 19
},
"restartEvents": [],
"restartHistory": [],
"status": "Success"
},
"doc_as_upsert": true
}
"__v": 2 in this mongodb data version I got success status but this version is not present it into AWS open search.
I tried so many thing but no luck, Below is my toml configuration currently I have
resume-from-timestamp = 1739336325
verbose = true
resume = true
replay = true
index-oplog-time=true
index-as-update = true
elasticsearch-max-docs = 1
elasticsearch-max-conns = 5
elasticsearch-max-seconds = 5
gtm-settings = {buffer-size = 1024, buffer-duration = "2s"}
I'm using Monstache 6 docker image with Javascript.
Can someone please help how to overcome this problem?
I tried to use external option but did not worked, May be I did something wrong!
Upvotes: 0
Views: 14