MongoDB: Aggregate Data Using Map/Reduce

Question

I still don't fully understand how map/reduce works, so I thought I'd give an example of a problem I need to solve, and hopefully the answer will help me understand the concept.

I'm tracking page views using a document structure similar to this:

{
    "timestamp" : 1299990045,
    "visitor" : {
        "region" : {
            "country_code" : "US",
        },
        "browser" : {
            "name" : "IE",
            "version" : "8.0",
        }
    },
    "referer" : {
        "host" : "www.google.com",
        "path" : "/",
        "query" : "q=map%2Freduce"
    }
}

I store a single document for each page view. Because I get about 15 million page views a day, I'd like to aggregate these results each night, save the aggregate results for that day, and then drop the collection to begin storing page views again. I want the output of the map/reduce to look like this:

{
    "day" : "Sun Mar 13 2011 00:00:00 GMT-0400 (EDT)",
    "regions" : {
        "US" : 235,
        "CA" : 212,
        "JP" : 121
    },
    "browsers" : {
        "IE" : 145,
        "Firefox" : 245,
        "Chrome" : 95,
        "Other" : 120
    },
    "referers" : {
        "www.google.com" : 24,
        "yahoo.com" 56
    }
}

I really don't know where to begin doing this kind of thing. Any help would be appreciated.

Answer 1

The typical process for writing a map-reduce job is to start with the data format you want as the output of your reduce, build a map function that outputs it, then a reduce function that adds them up. In your example, you'd do something like this:

function map() { 
    var date = new Date( this.timestamp.getFullYear(), 
                         this.timestamp.getMonth(),
                         this.timestamp.getDay() );
    var out  = { regions: {}, browsers: {}, referers: {} };
    out.regions[ this.visitor.region.country_code ] = 1; 
    out.browsers[ this.visitor.browser.name ] = 1;
    out.referers[ this.referer.host ] = 1; 
    emit( date, out);
}

function reduce( key, values ) { 
    var out = { regions: {}, browsers: {}, referers: {} };
    values.forEach(function(value) { 
      for( var region in value.regions ) { 
        if( out.regions[region] ) { 
          out.regions[ region ] += value[region];
        } else { 
          out.regions[ region ] = value[region];
        }
      };
      for( var browser in value.browsers ) { 
        if( out.browsers[browser] ) { 
          out.browsers[ browser ] += value[browser]; 
        } else { 
          out.browsers[ browser ] = value[browser]; 
        }
      };
      for( var referer in value.referers ) { 
        if( out.referers[ referer] ) { 
          out.referers[ referer ] += value[referer];
        } else {  
          out.referers[ referer ] = value[referer];
        } 
      }
    });
    return out; 
} 

At the end of this, you should have an output collection that looks something like this:

{ 
  _id: "Sun Mar 13 2011 12:23:58 GMT-0700 (PDT)", 
  value: { 
    regions:  {
      "US" : 235,
      "CA" : 212,
      "JP" : 121
    },
    browsers: { 
      "IE" : 145,
      "Firefox" : 245,
      "Chrome" : 95,
      "Other" : 120 
    }, 
    referers: { 
      "www.google.com" : 24,
      "yahoo.com" 56
    } 
  } 
}

Note that there's another way you can do this.. Rather than doing a map reduce job, you can also keep all of this data in real time using atomic increments and upserts.

For example, each time you generate one of your page view docs, you could also do an update like this:

db.pageviews.summaries.update( { _id: new Date( this.timestamp.getFullYear(), 
                                                this.timestamp.getMonth(),
                                                this.timestamp.getDay() ) },
                               { $inc : {
                                     'visitor.region.US' : 1,
                                     'visitor.browser.IE' : 1,
                                     'referer.www.google.com' : 1 
                                  }
                               },
                               true // upsert
                             );

This would mean that you always have your summary document up to date and don't need any map reduce jobs.

Note, you might want to escape the '.' in your domain names as Mongo will interpret that as a hierarchy of documents rather than an attribute name.