Reputation: 1
We need to add a field 'doc_creationtime' in "yyyy-mm-ddTHH:mm:ss.SSSZ" date or string format into every document inserted into MongoDb collection using pipeline. The date value should not be calculated in pipeline stages. It should be fetched from the Mongo DB at the exact time when the document is getting inserted in Mongo DB
As our requirement is to get the dcoument creation time till milliseconds, obtaining document creation time from 'Object_Id' in MongoDb cannot be used since its contains the dcoument creation time only till seconds.
we are already using the below to calculate the time, the record passes through the pipeline and assign to field 'docfetchedtime' ${time:extractStringFromDateTZ(time:now(), 'GMT', "yyyy-MM-dd'T'HH:mm:ss.SSS+00:00")}
The over all requirement is to get the differnce between document creation time in Mongo DB and the above calculated 'docfetchedtime' to get to know how much time pipeline is taking to fetch and load one document
we are already using the below to calculate the time, the record passes through the pipeline and assign to field 'docfetchedtime' ${time:extractStringFromDateTZ(time:now(), 'GMT', "yyyy-MM-dd'T'HH:mm:ss.SSS+00:00")}
The over all requirement is to get the differnce between document creation time in Mongo DB and the above calculated 'docfetchedtime' to get to know how much time streamsets pipeline is taking to fetch and load one document
Upvotes: 0
Views: 215
Reputation: 960
I use timestamps:true in the collection schema/model, like this:
const schema = new Schema({
Field_1:{type:String, trim:true, uppercase:true, required:true},
Field_2:{type:String, trim:true, lowercase:true, required:true},
...
},{
collection:'collectionName',
timestamps:true
})
This automaticly creates the fields createdAt and updatedAt in the document.
And I use the library moment.js to works with dates.
Upvotes: 0