CODEWITHSUNDEEP
streamingamazon-web-serviceshadoopmapreduce
aeolist
aeolist

Reputation: 196

Hadoop or Hadoop Streaming for MapReduce on AWS

I'm about to start a mapreduce project which will run on AWS and I am presented with a choice, to either use Java or C++.

I understand that writing the project in Java would make more functionality available to me, however C++ could pull it off too, through Hadoop Streaming.

Mind you, I have little background in either language. A similar project has been done in C++ and the code is available to me.

So my question: is this extra functionality available through AWS or is it only relevant if you have more control over the cloud? Is there anything else I should bear in mind in order to make a decision, like availability of plugins for hadoop that work better with one language or the other?

Thanks in advance

Upvotes: 5

Views: 1450

Answers (3)

aeolist
aeolist

Reputation: 196

I decided the flexibility of Java was more important than dealing with the possible shortcomings of adjusting my current code from C++ to Java.

Thanks for all your answers.

Upvotes: 1

Jeff Hammerbacher
Jeff Hammerbacher

Reputation: 4236

You have a few options for running Hadoop on AWS. The simplest is to run your MapReduce jobs via their Elastic MapReduce service: http://aws.amazon.com/elasticmapreduce. You could also run a Hadoop cluster on EC2, as described at http://archive.cloudera.com/docs/ec2.html.

If you suspect you'll need to write your own input/output formats, partitioners, and combiners, I'd recommend using Java with the latter system. If your job is relatively simple and you don't plan to use your Hadoop cluster for any other purpose, I'd recommend choosing the language with which you are most comfortable and using EMR.

Either way, good luck!

Disclosure: I am a founder of Cloudera.

Regards, Jeff

Upvotes: 6

sagie
sagie

Reputation: 3026

It depends on your needs. What is your input/output? Is it a simple text files? Records with new line delimiters? Do you need a special combiner? partitioner?

What i mean is, that if you need only the hadoop basics, than streaming will be fine. But if you need a little more complexity (from the hadoop framework, not from your own business logic), hadoop jar will be more flexible.

Sagie

Upvotes: 0

Related Questions