CODEWITHSUNDEEP
javascriptmongodbmeteor
samcorcos
samcorcos

Reputation: 2488

Meteor 1.0 - Mongo queries using variables as key, including $inc

I'm working with a large dataset that needs to be efficient with its Mongo queries. The application uses the Ford-Fulkerson algorithm to calculate recommendations and runs in polynomial time, so efficiency is extremely important. The syntax is ES6, but everything is basically the same.

This is an approximation of the data I'm working with. An array of items and one item being matched up against the other items:

let items = ["pen", "marker", "crayon", "pencil"];
let match = "sharpie";

Eventually, we will iterate over match and increase the weight of the pairing by 1. So, after going through the function, my ideal data looks like this:

{
  sharpie: {
    pen: 1,
    marker: 1,
    crayon: 1,
    pencil: 1
  }
}

To further elaborate, the value next to each key is the weight of that relationship, which is to say, the number of times those items have been paired together. What I would like to have happen is something like this:

// For each in the items array, check to see if the pairing already
// exists. If it does, increment. If it does not, create it.
_.each(items, function(item, i) {  
  Database.upsert({ match: { $exist: true }}, { match: { $inc: { item: 1 } } });
})

The problem, of course, is that Mongo does not allow bracket notation, nor does it allow for variable names as keys (match). The other problem, as I've learned, is that Mongo also has problems with deeply nested $inc operators ('The dollar ($) prefixed field \'$inc\' in \'3LhmpJMe9Es6r5HLs.$inc\' is not valid for storage.' }).

Is there anything I can do to make this in as few queries as possible? I'm open to suggestions.

EDIT

I attempted to create objects to pass into the Mongo query:

    _.each(items, function(item, i) {
        let selector = {};
        selector[match] = {};
        selector[match][item] = {};

        let modifier = {};
        modifier[match] = {};
        modifier[match]["$inc"] = {};
        modifier[match]["$inc"][item] = 1

        Database.upsert(selector, modifier);

Unfortunately, it still doesn't work. The $inc breaks the query and it won't let me go more than 1 level deep to change anything.

Solution

This is the function I ended up implementing. It works like a charm! Thanks Matt.

  _.each(items, function(item, i) {

    let incMod = {$inc:{}};
    let matchMod = {$inc:{}};

    matchMod.$inc[match] = 1;
    incMod.$inc[item] = 1;

    Database.upsert({node: item}, matchMod);
    Database.upsert({node: match}, incMod);
  });

Upvotes: 3

Views: 183

Answers (1)

Matt K
Matt K

Reputation: 4946

I think the trouble comes from your ER model. a sharpie isn't a standalone entity, a sharpie is an item. The relationship between 1 item and other items is such that 1 item has many items (1:M recursive) and each item-pairing has a weight.

Fully normalized, you'd have an items table & a weights table. The items table would have the items. The weights table would have something like item1, item2, weight (in doing so, you can have asymmetrical weighting, e.g. sharpie:pencil = 1, pencil:sharpie = .5, which is useful when calculating pushback in the FFA, but I don't think that applies in your case.

Great, now let's mongotize it.

When we say 1 item has many items, that "many" is probably not going to exceed a few thousand (think 16MB document cap). That means it's actually 1-to-few, which means we can nest the data, either using subdocs or fields.

So, let's check out that schema!

doc =
{
  _id: "sharpie",
  crayon: 1,
  pencil: 1
}

What do we see? sharpie isn't a key, it's a value. This makes everything easy. We leave the items as fields. The reason we don't use an array of objects is because this is faster & cleaner (no need to iterate over the array to find the matching _id). 

var match = "sharpie";
var items = ["pen", "marker", "crayon", "pencil"];
var incMod = {$inc:{}};
var matchMod = {$inc:{}};
matchMod.$inc[match] = 1;
for (var i = 0; i < items.length; i++) {
  Collection.upsert({_id: items[i]}, matchMod);
  incMod.$inc[items[i]] = 1;  
}
Collection.upsert({_id: match}, incMod);

That's the easy part. The hard part is figuring out why you want to use an FFA for a suggestion engine :-P.

Upvotes: 2

Related Questions