CODEWITHSUNDEEP
mongodbstreamsets
bhanvi
bhanvi

Reputation: 1

Load document creation time field in Mongo DB from streamsets pipeline

We need to add a field 'doc_creationtime' in "yyyy-mm-ddTHH:mm:ss.SSSZ" date or string format into every document inserted into MongoDb collection using pipeline. The date value should not be calculated in pipeline stages. It should be fetched from the Mongo DB at the exact time when the document is getting inserted in Mongo DB

As our requirement is to get the dcoument creation time till milliseconds, obtaining document creation time from 'Object_Id' in MongoDb cannot be used since its contains the dcoument creation time only till seconds.

we are already using the below to calculate the time, the record passes through the pipeline and assign to field 'docfetchedtime' ${time:extractStringFromDateTZ(time:now(), 'GMT', "yyyy-MM-dd'T'HH:mm:ss.SSS+00:00")}

The over all requirement is to get the differnce between document creation time in Mongo DB and the above calculated 'docfetchedtime' to get to know how much time pipeline is taking to fetch and load one document

we are already using the below to calculate the time, the record passes through the pipeline and assign to field 'docfetchedtime' ${time:extractStringFromDateTZ(time:now(), 'GMT', "yyyy-MM-dd'T'HH:mm:ss.SSS+00:00")}

The over all requirement is to get the differnce between document creation time in Mongo DB and the above calculated 'docfetchedtime' to get to know how much time streamsets pipeline is taking to fetch and load one document

Upvotes: 0

Views: 215

Answers (1)

Sergi Nadal
Sergi Nadal

Reputation: 960

I use timestamps:true in the collection schema/model, like this:

const schema = new Schema({
    Field_1:{type:String, trim:true, uppercase:true, required:true},
    Field_2:{type:String, trim:true, lowercase:true, required:true},
    ...
},{
    collection:'collectionName',
    timestamps:true
})

This automaticly creates the fields createdAt and updatedAt in the document.

enter image description here

And I use the library moment.js to works with dates.

Upvotes: 0

Related Questions