CODEWITHSUNDEEP
hadoopmapreduce
smrt28
smrt28

Reputation: 469

Reduce properties which I'm not sure about

I'm a beginner in writing map-reduces and I'm not sure about some reduce function properties.

So, reduce gets (key, list of values) as an input parameter...

  1. is it guaranteed that the list of input values always contains at least 2 members? So, an unique key emitted by the mapper would never be passed to the reducer?
  2. or, if there is just one item in the input list, is it guaranteed that the key is unique?
  3. can reduce emit more values then the input values list size?
  4. I have a large list of strings. I need to find all of them which are not unique. Can I make it with just one map/reduce? The only way I see is to count all the unique strings by one map/reduce and then select those which are not unique by the another map/reduce

Thanks

Upvotes: 0

Views: 38

Answers (2)

Chris Gerken
Chris Gerken

Reputation: 16390

The list of input values to the reduce() method may have one or more, but not zero members.

All of the values mapped from/to a unique key value are passed as a list to the reduce along with the key value. If that list contains one member then you can assume that that key value was mapped to only one value (or once, if you're counting)

Your reducer can write any number, including zero, of key value pairs for a given input key and list of values. The types of the input key/values may be different from the types of the output key/value pairs.

You can solve your problem with one map/reduce step

Upvotes: 1

smrt28
smrt28

Reputation: 469

So, the problem with the strings, pseudocode:

map(string s) {
    emit(s, 0);
}


reduce(string key, list values) {
    if (valies.size() > 1) { emit(key, 1); return; }
    if (valuse.contains(1)) { emit(key, 1); return; }
}

right?

Upvotes: 0

Related Questions