CODEWITHSUNDEEP
node.js
baldmark
baldmark

Reputation: 707

Processing a stream in Node where action depends on asynchronous calls

I am trying to write a node program that takes a stream of data (using xml-stream) and consolidates it and writes it to a database (using mongoose). I am having problems figuring out how to do the consolidation, since the data may not have hit the database by the time I am processing the next record. I am trying to do something like:

on order data being read from stream
  look to see if customer exists on mongodb collection 
    if customer exists 
      add the order to the document
    else
      create the customer record with just this order
    save the customer

My problem is that two 'nearby' orders for a customer cause duplicate customer records to be written, since the first one hasn't been written before the second one checks to see if it there.

In theory I think I could get around the problem by pausing the xml-stream, but there is a bug preventing me from doing this.

Upvotes: 2

Views: 198

Answers (2)

baldmark
baldmark

Reputation: 707

Not sure that this is the best option, but using async queue was what I ended up doing.

At the same time as I was doing that a pull request for xml-stream (which is what I was using to process the stream) that allowed pausing was added.

Upvotes: 1

Noah
Noah

Reputation: 34343

Is there a unique field on the customer object in the data coming from the stream? You could add a unique restriction to your mongoose schema to prevent duplicates at the database level.

When creating new customers, add some fallback logic to handle the case where you try to create a customer but that same customer is created by another save at the same. When this happens try the save again but first fetch the other customer first and add the order to the fetched customer document

Upvotes: 0

Related Questions