Support for Hive, Sqoop & Pig

Question

1. Does Apache Atlas support the tags for Spark, Pig, and Sqoop?

   • If Atlas support Pig, Sqoop & Spark then is there any kind of customization
     that we have to perform to make it work Atlas.
   • From where we can gets the exact process if we want to implement the above
     Hadoop components with Atlas.

2. Where does Atlas store the metadata?

   • How we can reach the schema of Atlas metadata.

3. Does Atlas provide the feature to fetch metadata for other tools (e.g. Informatica) and integrate it with its own metadata?

   • If Atlas support the above feature then how we can integrate the informatica with Atlas metadata.

Answer 1

Question 1: The list of bridges can be found at http://atlas.apache.org/index.html

At the time of answering, there are only bridges for hive, sqoop, falcon and storm.

To add Spark and Pig support, there are two options:

• Either use the Atlas Rest API(http://atlas.apache.org/api/v2/index.html) and trigger it in your code

• Implement our own bridge (https://github.com/apache/incubator-atlas/tree/master/addons)

Question 2: quoting http://atlas.apache.org/Architecture.html

Titan: Currently, Atlas uses the Titan Graph Database to store the metadata objects. Titan is used as a library within Atlas. Titan uses two stores: The Metadata store is configured to !HBase by default and the Index store is configured to Solr. It is also possible to use the Metadata store as BerkeleyDB and Index store as ElasticSearch by building with corresponding profiles. The Metadata store is used for storing the metadata objects proper, and the Index store is used for storing indices of the Metadata properties, that allows efficient search.

Question 3: See question 1, it would not be any different from an implementation for Spark or Pig.

Answer 2

Apache atlas support hive. Please check the below link

http://atlas.incubator.apache.org/Bridge-Hive.html

I hope this will help you. Not sure about your second and third question.