CODEWITHSUNDEEP
hadoophivemahoutmahout-recommender
user1219626
user1219626

Reputation:

what should be the correct flow of data in hadoop and mahout?

I am working with hadoop, hive and mahout technology. I am processing some data with a mapreduce job in hadoop for recommendation purposes in mahout. I want to know the correct workflow of above model, i.e when hadoop processes the data and stores it in HDFS, then how will mahout use this data and how will mahout get this data and after mahout processes the data, where will mahout put this recommended data? Note: I am working with hadoop for processing the data and my colleague is working with mahout on a different machine . Hope u got my question correctly.

Upvotes: 1

Views: 143

Answers (2)

user1261215
user1261215

Reputation:

It all depends on how Mahout is configured to run. Mahout can run in local mode or distributed mode. We need to set the "MAHOUT_LOCAL" variable.

  MAHOUT_LOCAL       set to anything other than an empty string to force
                      mahout to run locally even if
                      HADOOP_CONF_DIR and HADOOP_HOME are set

For example, If we don't configure MAHOUT_LOCAL and tries to execute any Mahout algorithm, Then you can see below in the console.

MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Running on hadoop,

When running in distributed mode, Mahout treats all the paths as HDFS path's. So even after Mahout processing your data, final output will be stored in HDFS.

Upvotes: 0

Kishore
Kishore

Reputation: 5881

If you want to take input from hadoop hdfs in mahout then you have to do following steps-

first copy input file to hdfs by command

hadoop dfs -copyFromLocal input /

Then run the mahout command for recommendation which take input from hdfs and save the output in hdfs

Assuming your JAVA_HOME is appropriately set and Mahout was installed properly we’re ready to configure our syntax. Enter the following command:

$ mahout recommenditembased -s SIMILARITY_LOGLIKELIHOOD -i hdfs://localhost:9000/inputfile -o hdfs://localhost:9000/output --numRecommendations 25

Running the command will execute a series of jobs the final product of which will be an output file deposited to the directory specified in the command syntax. The output file will contain two columns: the userID and an array of itemIDs and scores.

Upvotes: 2

Related Questions