CODEWITHSUNDEEP
fiware-orionfiwarefiware-cygnus
user3698602
user3698602

Reputation:

Fi-Ware Bigdata configuration

I'm reading a tutorial to Bigdata installationg over Fi-Ware.

https://github.com/telefonicaid/fiware-connectors/blob/develop/flume/README.md

I've completed susccesfully steps up to "Installing Cygnus and its dependencies". I've realized there is a new Cygnus version (0.3 instead of 0.2.1 used in the tutorial). I'd like to know if it is necesary to acomplish another step related to dependences installation.

I found in the tutorial the next paragraph

OrionCKANSink dependencies

These are the packages you will need to install under APACHE_FLUME_HOME/plugins.d/cygnus/libext if you did not included them in the Cygnus package:

json-simple-1.1.jar
OrionHDFSSink dependencies

These are the packages you will need to install under APACHE_FLUME_HOME/plugins.d/cygnus/libext if you did not included them in the Cygnus package:

hadoop-core-0.20.0.jar (or higher)
hive-exec-0.12.0.jar
hive-jdbc-0.12.0.jar
hive-metastore-0.12.0.jar
hive-service-0.12.0.jar
hive-common-0.12.0.jar
hive-shims-0.12.0.jar

These packages are not necessary to be installed since they are already included in the Flume bundle (they have been listed just for informative purposes):

httpclient-4.2.1.jar
httpcore-4.2.2.jar

In addition, as already said, remember to overwrite the APACHE_FLUME_HOME/lib/libthrift-0.7.0.jar package with this one:

libthrift-0.9.1.jar

Finally, if you are planning to use the OrionMySQLSink, include the latest MySQL connector in APACHE_FLUME_HOME/plugins.d/cygnus/libext:

mysql-connector-java-5.1.26-bin.jar

Is there an easier way to install and configure cygnus?? It seems to difficult to use Bigdata with fi-ware.

Thank you

Upvotes: 2

Views: 358

Answers (1)

frb
frb

Reputation: 3798

First of all, some important remarks:

Being said that, installing Cygnus is quite simple. Basically, you have to install Apache Flume and then build the Cygnus jar which will contain additional classes for Flume.

Cygnus 0.1 building is as simple as doing:

$ cd /path/to/cygnus/folder
$ /path/to/maven/bin/mvn package
$ cp target/cygnus-0.1.jar /path/to/flume/lib

Nothing else is required since such version has no support for MySQL nor Hive tables automatic creation.

Cygnus 0.2 and Cygnus 0.2.1 both require several third-party libraries, but most of them are pacakged with the Cygnus classes themselves by using this command, as documented in the README:

$ cd /path/to/cygnus/folder
$ /path/to/maven/bin/mvn clean compile assembly:single
$ cp target/cygnus-0.2.x-jar-with-dependencies.jar /path/to/flume/plugins.d/cygnus/lib

The only additional thing you have to do is to upgrade the Flume libthrift library and to copy the MySQL connector in /path/to/flume/plugins.d/cygnus/libext. The list of libraries you've found in the README are listed just for documentation purposes (as stated there).

Anyway, we are working on the generation of RPMs automating the above steps. So stay tuned!

Upvotes: 0

Related Questions