CODEWITHSUNDEEP
couchbase
user1978109
user1978109

Reputation: 737

Importing large datasets into Couchbase

I am having difficulty importing large datasets into Couchbase. I have experience doing this very fast with Redis via the command line but I have not seen anything yet for Couchbase.

I have tried using the PHP SDK and it imports about 500 documents / second. I have also tried the cbcdocload script in the Couchbase bin folder but it seems to want each document in its on JSON file. It is a bit of work to create all these files and then load them. Is there some other importation process I am missing? If cbcdocload is the only way load data fast then is it possible to put multiple documents into 1 json file.

Upvotes: 1

Views: 2686

Answers (1)

Austin Gonyou
Austin Gonyou

Reputation: 252

Take the file that has all the JSON documents in it and zip up the file: 

zip somefile.zip somefile.json

Place the zip file(s) into a directory. I used ~/json_files/ in my home directory.

Then load the file or files by the following command:

cbdocloader -u Administrator -p s3kre7Pa55 -b MyBucketToLoad -n 127.0.0.1:8091 -s 1000 \
~/json_files/somefile.zip

Note: '-s 1000' is the memory size. You'll need to adjust this value for your bucket.

If successful you'll see output stating how many documents were loaded, success, etc.

Here is a brief script to load up a lot of .zip files in a given directory:

#!/bin/bash
JSON_Dir=~/json_files/
for ZipFile in $JSON_Dir/*.zip ; 
do /Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbdocloader  \
    -u Administrator -p s3kre7Pa55 -b MyBucketToLoad                                  \
    -n 127.0.0.1:8091 -s 1000 $ZipFile
 done

UPDATED: Keep in mind this script will only work if your data is formatted correctly or if the files are less than the max single document size of 20MB. (not the zipfile, but any document extracted from the zip)

I have created a blog post describing bulk loading from a single file as well and it is listed here:

Bulk Loading Documents Into Couchbase

Upvotes: 1

Related Questions