SQL Server Database Management with Continuous Integration

Question

Let's say we have a continuous integration server. When I check in, the post-hook pulls the latest code, runs the tests, packages everything. What is the best way to also automate the database changes?

Ideally, I'd build an installer that could either build a database from scratch or update an existing one using some automated syncing method.

Answer 1

I've recently bumped into an article, that might be of use.

The author explained some of the best continuous integration practices including testing, processing and automation.

Here are some of the key takeaways:

• In many shops code is unit tested at the point of commit. For databases, it is preferred running all unit tests at once and in sequence against a QA database, vs development, as a part of the Test step
• The test step is a critical part of any CI/CD process. Test scripts, including unit tests themselves, should also be versioned in source control, extracted at the point of the Build step and executed
• Pulling data from production is appealing as a quick expedient, but is never a good idea
• The best approach is using a tool or script to quickly, repeatedly and reliably create synthetic test data for your transactional tables
• Running unit tests to produce manual summary results for human consumption defeats the purpose of automation. We need machine readable results, that can allow an automated process to abort, branch and/or continue.
• Running a CI process, which requires 100% of all tests to pass, is akin to not having CI at all, if the workflow pipeline is set up atomically to stop on failure, which it should. To thread the needle, tests should have built in thresholds, that will raise an error based on either the % of tests failing or in some cases, if certain high priority tests fail.
• All processes should ultimately produce a Boolean result of pass or fail, but some non-automated processes can easily find their way into your CI workflow pipeline (e.g. unit testing). Software should be plug-n-play into any workflow pipeline, taking known inputs and producing expected outputs – like pass, fail.
• CI/CD process should be aborted on failure and a notification email should be immediately sent vs continuing to cycle the pipeline.
• The CI process should not cycle again until any errors in the last build are fixed. On failure, the entire team should get the failure notification, including as many details as to what failed as possible.
• If a pipeline takes 1 hour, from start to finish, to complete, including all the testing, then all the build intervals should be set to no less than one hour and all new commits should be queued, and applied to the next build.
• No plain text passwords should exist in automation scripts

Answer 2

I know this post is old, but we have a new solution that takes the following approach:

1. Developers script individual SQL changes and commit them to source control.
2. Our program (OneScript) pulls the change script files from source control, filters and sorts them, and generates a single release script file.
3. That release script file is then applied to a database to do a release.

Our home page here explains this process in more detail and has a link to an example that does these steps automatically from a Subversion hook. So soon after a commit, the developer receives an email saying if the release was successful or had errors. The PowerScript code is included.

Disclaimer -I'm working at the company that makes OneScript.

Answer 3

Have you looked at FluentMigrator? The default download includes Nant scripts that would be easy to add in to a CI. Free, open source and easy to use. Works for a wide variety of databases.

Answer 4

The Red Gate approach using SQL Source Control and the SQL Compare Pro command line is detailed with code samples here: http://downloads.red-gate.com/HelpPDF/ContinuousIntegrationForDatabasesUsingRedGateSQLTools.pdf

Troy Hunt wrote an article on Simple Talk entitled "Continuous Integration for SQL Server Databases": http://www.simple-talk.com/content/article.aspx?article=1247

Answer 5

If you have the opportunity to define and control the whole database management and db creation process, have a serious look at DB Ghost - it's more than just a tool - it's a process.

If you like it and can implement it, you'll get great returns on it - but it's a bit of a "all-or-nothing" kind of approach. Recommended.

Answer 6

Red Gate is a quite robust solution and it works out of the box. But the best thing is that you can integrate it with your continuous integration process. I use it with Msbuild and Hudson. quickly explaining how it works: http://blog.vincentbrouillet.com/post/2011/02/10/Database-schema-synchronisation-with-RedGate

if you need to know more about this, feel free to ask

Answer 7

I would caution against using a db backup as a development artifact, most CI best practices suggest that you manage the schema, procedures, triggers, and views as first class development artifacts. The side effects is that you can take this one step further and use them to build a new database whenever you want, ideally you also have some data that can be pushed into the database.

Here is a cliff notes version to get your feet wet, but there is lots out there in this space: http://www.infoq.com/news/2008/02/versioning_databases_series

I like some of the ideas that Scott Ambler has here as well, the site is good but the book is surprisingly deep for such a difficult set of problems. http://www.agiledata.org/ http://www.amazon.com/exec/obidos/ASIN/0321293533/ambysoftinc

Answer 8

The latest version (5.0) of DB Ghost doesn't suffer from the "non ASCII character" problem (it just means that the file is UTF8 encoded) and it should be able to do exactly what you need.

Also, the tools can actually be used standalone to perform the various functions (scripting, building, comparing, upgrading and packaging) if you want, it's just that using them all together provides a full end-to-end process thus making the overall value greater than the sum of it's parts.

In essence, to make changes to the schema you update individual object creation scripts and per-table insert scripts (for reference data) that are held under source control just like you were developing a “day one” greenfield database. The DB Ghost tools are used to enable the whole thing by building these scripts into a brand new database (using continuous integration if required) and then comparing and upgrading a target database, which can be a copy of the production database. This process produces a delta script which can be used on the real production database during go-live.

You can even produce a Visual Studio database project and add it into any solutions you currently have.

Malc