CODEWITHSUNDEEP
mysqlmahout
qian qian
qian qian

Reputation: 281

Mahout 0.7 Failed to get recommendation with a large data using MysqlJdbcDataModel

I am using Mahout to build an Item-based Cf recommendation engine. I create an MahoutHelper class which has a constructor:

    public MahoutHelper(String serverName, String user, String password,
        String DatabaseName, String tableName) {


    source = new MysqlConnectionPoolDataSource();

    source.setServerName(serverName);
    source.setUser(user);
    source.setPassword(password);
    source.setDatabaseName(DatabaseName);
    source.setCachePreparedStatements(true);
    source.setCachePrepStmts(true);
    source.setCacheResultSetMetadata(true);
    source.setAlwaysSendSetIsolation(true);
    source.setElideSetAutoCommits(true);
    DBmodel = new MySQLJDBCDataModel(source, tableName, "userId", "itemId",
            "value", null);

    similarity = new TanimotoCoefficientSimilarity(DBmodel);

}

and the recommend method is:

   public List<RecommendedItem> recommendation() throws TasteException {

    Recommender recommender = null;
    recommender = new GenericItemBasedRecommender(DBmodel, similarity);
    List<RecommendedItem> recommendations = null;
    recommendations = recommender.recommend(userId, maxNum);
    System.out.println("query completed");
    return recommendations;
}

It's using datasource to build datamodel but the problem is that when mysql has only a few data (less than 100) the program works fine for me, while when the scale turns to be over 1,000,000, the program stacks at doing recommendation and never goes forward. I have no idea how it happens. By the way I used the same data to build a FileDataModel with a .dat file, and it takes only 2~3 second to complete analysis. I am confused.

Upvotes: 1

Views: 328

Answers (1)

Sean Owen
Sean Owen

Reputation: 66876

Using the database directly will only work for tiny data sets, like maybe a hundred thousand data points. Beyond that the overhead of such data-intensive applications will never run quickly; a query takes thousands of SQL queries or more.

Instead you must load and re-load into memory. You can still pull from the database; look at ReloadFromJDBCDataModel as a wrapper.

Upvotes: 2

Related Questions