CODEWITHSUNDEEP
jsoncouchbasenosql
Tismon Varghese
Tismon Varghese

Reputation: 869

Load JSON file to couchbase server

I have JSON file with 1.5GB size and i need to upload this to a couchbase server bucket. I use the following command to load.

cbdocloader -n 10.5.2.117:8091 -u Administrator -p password -b mybucket ../samples/gamesim.zip

Unfortunately, it beaks because of memory issue. Can someone suggest a better method or an optimal way to solve this ? My machine is having 15GB memory and Intel Xeon CPU with 2.50GHz speed.

Thanks in advance, Tismon Varghese.

Upvotes: 1

Views: 1103

Answers (1)

Amol
Amol

Reputation: 344

I used a small java program to implement this. You can refer this link http://architects.dzone.com/articles/sql-nosql-copy-your-data-mysql Migrated one of the large single MySQL databse table(28Lac records) using this(you can migrate entire database using this program, even all databases ).

MySQL OS : Ubuntu 14.04 (Amazon Instance) Memory : 7.4G

Couchbase OS : Ubuntu 14.04 (Amazon Instance) Memory : 15G (2xlarge instance) Java : java version "1.7.0_79"
Note: Memory eaten up while using a Amazone machine with Memory 7.4G

Step 1. Download couchbase-sql-importer from github (https://github.com/tgrall/couchbase-sql-importer) to Couchase server, some where in your home folder and extract it.

Step3. Download 'mysql-connector-java-5.1.25-bin.jar' from http://downloads.mysql.com/archives/c-j/ to the 'couchbase-sql-importer-master' (extrated folder). While dowloding jar file from mysql site, you need to select the jar fie version from drop down.

Step4.Download 'CouchbaseSqlImporter.jar' from https://www.dropbox.com/s/pcnwukw2j8xe2d2/CouchbaseSqlImporter.jar to the folder 'couchbase-sql-importer-master'.

Step5.Edit the fie 'import.properties' like, 

## SQL Information ##
sql.connection=jdbc:mysql://mysql_server_ip:3306/database_name
sql.username=mysql_user_name
sql.password=myql_password

## Couchbase Information ##
cb.uris=http://localhost:8091/pools
cb.bucket=bucket_name
cb.password=bucket's_password

## Import information
import.tables=specific_table_name
import.createViews=true
import.typefield=
import.fieldcase=none

Save this file.

Important things to notice

cb.password=bucket's_password     
// you can set passwords for each bucket via Couchbase console. Select bucket and edit the feild 'Access Control' and give password in there and save.



import.tables=All  //For migrating entire table.

If import.properties file not found in 'couchbase-sql-importer-master' folder, then search and find sample import.properties file and rename it to so.

Step6. Run this command in terminal,

java -cp "./CouchbaseSqlImporter.jar:./mysql-connector-java-5.1.25-bin.jar" com.couchbase.util.SqlImporter import.properties

Output

############################################
#         COUCHBASE SQL IMPORTER           #
############################################



Importing table(s)
    from :  jdbc:mysql://XXXX:3306/XXX
    to :    [http://localhost:8091/pools] - XXXX

  Exporting Table : XXXX


 Create Couchbase views for XXXX
    2801937 records moved to Couchbase.


 Create Couchbase views for 'types' ....


              FINISHED
############################################

Check Couchbase console for verifying  the content.

Hope this is crystal clear for you :) Thanks.

Upvotes: 2

Related Questions