CODEWITHSUNDEEP
javaunit-testingmavenhadoophive
Avi
Avi

Reputation: 21760

Starting a simple standalone hadoop server in unit-test

Background: I'm trying to test a Hive query as part of our testing framework. I want to create short tests that test a small subset of the data in order for the queries to return fast, and be able to run them on every computer (even private laptops). The goal is to be able to just checkout the code, build using maven and run the tests.

Question: Is there a way for me to start a standalone hadoop (or some sort of simulator) only using java code, without downloads and installations that can be ran as part of the tests?

My goal is to have a test that in its @Before method set ups hadoop inside the tests which is deleted at the end of the test.

Upvotes: 1

Views: 675

Answers (2)

Nilkamal Weerasinghe
Nilkamal Weerasinghe

Reputation: 21

Take a look at HiveQLUnit - A open source project to unit test Hive.

  • Fast - It runs in memory through the use of HiveContext.
  • Standalone - Can run from your laptop/dexktop without server installations.
  • Maven Integration - Included in maven central and can be included easily as a maven dependency within your project.
  • JUnit Integration - Fully integrates with the JUnit unit testing framework

It also solves your problem of executing standalone unit tests from your favorite IDE.

It can execute unit tests from your favorite IDE on any operating system including Windows, Linux and Mac OSX. Here is some guidance around using HiveQLUnit - Documentation.

Note: I am affiliated with the HiveQLUnit OS project.

Upvotes: 1

Karthik Ramachandran
Karthik Ramachandran

Reputation: 12175

Have you looked at the MiniCluster apis (http://wiki.apache.org/hadoop/HowToDevelopUnitTests) ?

Here is a blog post that covers how to use the this API is some detail : http://www.lopakalogic.com/articles/hadoop-articles/hadoop-testing-with-minicluster/

It looks like what you might be looking for.

Upvotes: 2

Related Questions