CODEWITHSUNDEEP
hdfsetl
systemdebt
systemdebt

Reputation: 4941

How to pull data from API and store it in HDFS

I am aware of flume and Kafka but these are event driven tools. I don't need it to be event driven or real time but may be just schedule the import once in a day.

What are the data ingestion tools available for importing data from API's in HDFS?

I am not using HBase either but HDFS and Hive.

I have used R language for that for quite a time but I am looking for a more robust,may be native solution to Hadoop environment.

Upvotes: 0

Views: 1707

Answers (1)

Jared
Jared

Reputation: 2954

Look into using Scala or Python for this. There are a couple ways to approach pulling from an API into HDFS. The first approach would be to write a script which runs on your edge node(essentially just a linux server) and pulls data from the API and lands it in a directory on the linux file system. Then your script can use HDFS file system commands to put the data into HDFS.

The second approach would be to use Scala or Python with Spark to call the API and directly load the data into HDFS using a Spark submit job. Again this script would be run from an edge node it is just utilizing Spark to bypass having to land the data in the LFS.

The first option is easier to implement. The second option is worth looking into if you have huge data volumes or an API that could be parallelized by making calls to muliple IDs/accounts at once.

Upvotes: 2

Related Questions