CODEWITHSUNDEEP
javagoogle-app-enginememory-managementmemcachedmahout-recommender
Dries De Rydt
Dries De Rydt

Reputation: 808

App engine: How does memory work? (using Mahout with the app engine)

I am trying to use Mahout with the app engine. There are a few issues that arose but my main issue right now is: How can I make a recommendation if the model is too large to keep into memory?

I am using the memcache, but the Mahout datamodel is not serializable by default. I am serializing the FastIDSet instead, but then when I have to make a recommendation I have to get the FastIDSet from the memcache, use it as a parameter for the recommender, and then make the recommendation.

This would be fine but keeping both into memory gives me a heap space error, even though my database is not too large (19 MB).

I cannot do this in chunks because Mahout does expect to be initialized with a full data model.

Does anyone have experience with this problem?

Thanks and good day.

Upvotes: 4

Views: 96

Answers (2)

pferrel
pferrel

Reputation: 5702

You are using some old, on the road to depreciation, technology. The new Mahout correlation engine, called SimilarityAnalysis.cooccurrence is used in an end-to-end recommender under Apache 2 license. Check out the Universal Recommender, which is based on the Spark Mahout work and uses a search engine for serving recommendations

BTW I'm a Mahout committer working on this, not trying to disparage Mahout's older recommenders for no reason :-)

Upvotes: 0

Dries De Rydt
Dries De Rydt

Reputation: 808

I figured it out, had to set my scaling in the appengine-web.xml.

Still seeing about the speed of the recommender if I have to rebuild the model from the fastIDSET too often.

Upvotes: 1

Related Questions