CODEWITHSUNDEEP
javamysqlhadoop
Raju Sharma
Raju Sharma

Reputation: 2516

How to integrate ?Hadoop with Mysql

i have just started working with hadoop. i did some basic application like word coutnt. but now i need to tranfer data between Hadoop , HDFS and Mysql Database.

i tried following code but not getting correctly.

DbInputWritable.java

public class DBInputWritable implements Writable, DBWritable
{
   private int id;
   private String name;

   public void readFields(DataInput in) throws IOException {   }

   public void readFields(ResultSet rs) throws SQLException
   //Resultset object represents the data returned from a SQL statement
   {
     id = rs.getInt(1);
     name = rs.getString(2);
   }

   public void write(DataOutput out) throws IOException {  }

   public void write(PreparedStatement ps) throws SQLException
   {
     ps.setInt(1, id);
     ps.setString(2, name);
   }

   public int getId()
   {
     return id;
   }

   public String getName()
   {
     return name;
   }
}

DBOutputWritable.java

public class DBOutputWritable implements Writable, DBWritable
{
   private String name;
   private int count;

   public DBOutputWritable(String name, int count)
   {
     this.name = name;
     this.count = count;
   }

   public void readFields(DataInput in) throws IOException {   }

   public void readFields(ResultSet rs) throws SQLException
   {
     name = rs.getString(1);
     count = rs.getInt(2);
   }

   public void write(DataOutput out) throws IOException {    }

   public void write(PreparedStatement ps) throws SQLException
   {
     ps.setString(1, name);
     ps.setInt(2, count);
   }
}

pls any body help.

Upvotes: 1

Views: 697

Answers (4)

anyoneking
anyoneking

Reputation: 546

Maybe you can show us detail of the exception. Do you have the mysql jdbc driver in your Hadoop lib? by the way, sqoop is a very good tool for transferring data between Hadoop and a DB. If your dbdata is not very large, you can use it.

Upvotes: 1

Ashish
Ashish

Reputation: 51

Here is the working example for this to get connected hadoop with my sql database. Its really helping have a look:

http://javatute.com/javatute/faces/post/hadoop/2014/connect-mysql-with-hadoop.xhtml

Upvotes: 0

gasparms
gasparms

Reputation: 3354

I don't know exactly what's wrong in your code but have you tried Sqoop?

Sqoop is really easy importing data from sql databases to hdfs and vice versa. Once you have data in hdfs you don't have to implements any custom input.

Edit: You don't need an integration with Hadoop, just load data from your mysql database using sqoop and once you have data in hdfs, code your mappers and reducers setting the path of your files on hdfs within the Job Configuration. You can follow multiple mapreduce tutorials, you just have to change the files to read from to your files previously imported with Sqoop.

Upvotes: 1

Piyush Jain
Piyush Jain

Reputation: 63

You can HBQL For transferring data to hadoop you can find sample code example in link that given below

http://www.hbql.com/

thanks Piyush jain

Upvotes: 0

Related Questions