Application upgrade in a high availability environment

Question

I am writing a NoSQL database engine and I want to provide features to help the developers to upgrade their application to a new version without stopping the operation of the website, i.e 0% downtime during upgrade. So my question is, what are the methods or general design of a web application when it is run 24/7 and is changing its database structure very often? Any examples or success stories would be greatly appreciated.

Answer 1

The only possible case where this can be achieved is if you have a completely stateless application. The term stateless includes both application data and application structure. Remember the upgrade may involve changing the definition of the business objects in addition to data. Given that such stateless application is not practical due to obvious reasons, there is no practical way to achieve zero downtime for general upgrades. Any application that is not stateless will have the live users caching (in the mid-tier) business object definitions and business data. An upgrade may warrant not only new business data but new business object definitions. The cached data by live users can always cause potential inconsistencies with the new schema. So live users can not be migrated unless you can migrate both the data and meta data (business definitions) cached in the mid-tier. If you blow away the mid-tier cache, the live users will get affected. You may consider allowing live users to continue working under the old db and migrating/ merging any data changes to the new db later. But that is a complicated problem to solve as well. Now it is possible to restrict what is allowed under a zero downtime upgrade without affecting live users or make sure that after the database is upgraded, the live users just become read-only users unless they log out and log back in against the new schema.

Answer 2

When many webservers in a production environment access that database, and you have a incompatible code change (which removes a field and adds a new field), then I would advise the multi-step solution. It is a bit of work but you don't risc downtimes when some detail goes wrong.

First Step extend the application so that the old and the new version gets written, deploy that version

Second Step convert as far as possible the old data field values to the new data field (may take some time).

Third Step change the application to read only the new field, deploy it

Fourth Step remove the old field values

Fifth Step remove the writing of the old field values from code, deploy it.

Answer 3

If an enterprise can invest in geographical distribution of database. Like failover tolerance; it sounds traditional but data replication (or datastore replication) wouldn't be an issue for routing traffic.

Option 2:- use of caching (custom develop) & cycling. ex:- 1 am to 2am use snapshot 1 of database (let's say server1 /data center 1) 1:59am server2/data center 2 consists of new database architecture (new fields, new tables etc.,) and @ 2am all traffic route through data center 2.

Cycling basing the snapshot may be a solution to consider.

Answer 4

You're going to be building a distributed system. There's no way around this, as you'll need multiple machines involved to deal with things like reboots.

Building a distributed system means you're making some choices. Pick 2 of:

1. Durability
2. Availability
3. Strong Consistancy

Systems like S3, have chosen 1&2 and paid the price by sacrificing #3 in favor of "Eventual Consistancy". There's a great paper on S3 you can read. Other database solutions, like DynamoDB have chosen different trade offs.

You're going to need load balancers. Otherwise you're stuck with customers connecting directly to your service, which is rough for a variety of reasons. A Load Balancer lets you reboot a machine in your fleet without incurring down time. Reboots, as we all know, are a fact of life.

Doing what you're describing is very tough. In fact, I would say it's nearly an impossible problem for a single developer to tackle.

You are far, far, far more likely to get better results using an existing NoSQL database and spending all your time working on your product....

Answer 5

With NoSQL - and specifically a document oriented database - you can accomplish this with versioning.

Consider MongoDB, which stores everything as documents.

MongoDB allows you to have a collection (a group of documents) where the schema for every document can be different.

Let's say you have this document for a user:

{ "_id" : 100, "firstName" : "John", "lastName" : "Smith" }

You could also have this as a document in the same collection:

{ "_id" : 123, "firstName" : "John", "lastName" : "Smith", "hasFoo" : false }

Different schemas, but both in the same collection. Obviously this is very different from a traditional relational database.

The solution then is to add a field to every document that has the schema version. Then you have your application look for that version with every query.

A MongoDB query might look like this:

users.find({ "version" : 3 }).limit(10);

That just returns all users that are using schema version "3". You can insert newer schemas without affecting the existing site and slowly delete old schema versions that aren't useful anymore.

Answer 6

The database structure is closely related to the business rules, so the only scenario I can think of in which the database is changing often is in the development phase of a project.

In a production environment, it is assumed the application is already stable in terms of business rules, so changes to the structure of database are assumed to be rare. Therefore I think it will be very hard to find elaborate solutions for this case.

There is of course naive approaches like making an exact copy of the database before the upgrade, switching the application to run on the copy, upgrading and then switching back.

Other than this, I cannot think of anything else.