Pipeline filtering and processing with lodash

Question

I have the following lodash pseudocode, that would process auctions:

var auctions = [{'item': 12345, ...}, {'item': 12345, ...}, {'item':23436, ...}];

var process = function(auctions) {
  _(auctions)
    .groupBy('item')
    .thru(processSpecialItems)
    .each(processCommonItems)
    .value();
}

Each function depends on the result of the previous one, but in the case of processSpecialItems and processCommonItems:

• Only specific items, ie. the group identified by a given item value, would be processed by processSpecialItems.
• Only those groups who have not been processed should proceed to processCommonItems.
• These ops are CPU intensive and involve DB access.

Questions:

• How can I filter these special items, process them, and return the rest? Preferably with as less passes as possible, since data contains in excess of 40,000 items.
• Is it acceptable, performance wise, to use synchronous iteration? This snippet would be executed up to 400 times (all at the same time).

Answer 1

Assuming you are able to determine ahead of time which item groups would be handled by processSpecialItems, this is probably a job for _.partition.

From the docs:

Creates an array of elements split into two groups, the first of which contains elements predicate returns truthy for, the second of which contains elements predicate returns falsey for.

_.partition(auctions, isSpecial) would return [specialItems, commonItems] (where specialItems and commonItems are appropriate arrays of items). Then you can just have processSpecialItems operate on the first element and processCommonItems operate on the second.

Answer 2

dont group items at first step, do next

var process = function(auctions) {
    return _(auctions)
    .partition(function(obj) {
        // use here partition condition, 
        // which return true for special item,
        // something like 
        return _.startsWith(obj.item, 's');
    })
    // you receive array of 2 arrays where
    // first array - array of special items
    // second array - array of common items
    .thru(function(array){ // process items
        return [
            _.map(array[0], functionToProcessSpecialItem),
            _.map(array[1], functionToProcessCommonItem)
        ];
    })
    .flatten() // flat result
    .groupBy('item') // and group
    .value(); // you receive grouped and processed auctions
}