CODEWITHSUNDEEP
node.jscouchdbcloudant
Jed Richards
Jed Richards

Reputation: 12437

Moving data between production and staging databases on Cloudant

I have two databases on Cloudant, staging and production, which are being consumed by a Node.JS based web app running on Heroku, also with staging and production instances. The web app includes an admin area where admins can check on database content and tweak values etc.

What's the best approach for moving data between the two environments? Sometimes we'll want to move new values from staging to production, and perhaps other times move production data to staging to aid with testing etc., so it has to be two way.

Whole database replication is not an option since generally we'll only want to copy certain database objects, not the whole thing.

What sort of things should I be looking at to get this done in Node.JS? Any general approaches? Are we thinking about the setup wrong?

Upvotes: 3

Views: 739

Answers (1)

Sam Bisbee
Sam Bisbee

Reputation: 4441

There are two ways to do this with the database itself (versus writing client code to move data around, which would not be optimal).

For an up to date reference on replication, see http://wiki.apache.org/couchdb/Replication (Apache's wiki is down at the time of writing this answer).

Named Document Replication

If you know exactly which documents you want to replicate and their _id's, then you can specify them in an array when invoking the replication. For example, this would replicate the two docs with _id's doc-a and doc-b.

{
  "source": "https://USER:[email protected]/dbname-qa",
  "target": "https://USER:[email protected]/dbname-production",
  "doc_ids": [ "doc-a", "doc-b" ]
}

Filtered Replication

If you can programatically tell which records you want to move, then you can use filtered replication. You write a function that returns true if you want the doc to be replicated or false if you don't. For example...

function(doc) {
  /*
   * This will not replicate doc deletions. Add `&& doc._deleted` to replicate
   * those as well.
   */
  if(doc.type === "user") {
    return true;
  }

  return false;
}

...would replicate all the records whose type attribute is equal to "user".

You could also remove the whole if block and just do return (doc.type === "user"); if you are comfortable with it.

This code is stored in your design document. Here is an example ddoc that stores the above function:

{
  "_id": "_design/app",
  "filters": {
    "myfilter": "function(doc) { return (doc.type === \"user\"); }"
  }
}

You then invoke the filter during replication like this:

{
  "source": "https://USER:[email protected]/dbname-qa",
  "target": "https://USER:[email protected]/dbname-production",
  "filter": "app/myfilter"
}

Upvotes: 4

Related Questions