CODEWITHSUNDEEP
databasegitmigrationgerrit
TV's Frank
TV's Frank

Reputation: 818

Iterative database migration in with git/gerrit

Problem: how to handle sequential database migration scripts with slow commits / gerrit.

Setup is as follows:

The main issue with this approach (which works nicely otherwise) has been that when two developers add a migration script at the same time, the last of them to commit will get a merge conflict. In subversion (which we've used until now), this was a tree conflict which the commiter handled by reverting that file and adding it with a new filename, as the last action. Since we worked with this stuff on trunk, they would usually manage to fix the filename as a last quick action.

Now that we are migrating to git, the conflict will instead be shown as a diff inside the file, making it trickier to separate the two committed files (who may be part of different commit packages, along with other files). Furthermore, since we are using gerrit to code review, there will be a delay between pushing and getting the files into the main git repository, resulting in this scenario every now and then:

  1. You push a migration
  2. Gerrit gets a conflict after code review
  3. You fix the conflict by renaming the file, and push a new commit (which must be reviewed anew)
  4. Someone managed to get a new migration in before this was code reviewed, so Gerrit gets a conflict again
  5. You fix the conflict by renaming the file, and push a new commit (which must be reviewed anew)
  6. repeat from 4 until you are lucky and Gerrit manages to submit the migration successfully.

What is the best way to solve this situation?

Upvotes: 1

Views: 455

Answers (3)

bentolor
bentolor

Reputation: 3426

I second Liquibase as one solution. One property of Liquibase which Mark O'Connor IMHO didn't point out very well is, that Liquibase maintains a list of applied changesets vs. the last applied change in your approach.

Therefore (as long as the change identifiers do not clash) branches are not the problem. Liquibase simply applies all outstanding changeset which yet have not been applied in their sequential order within the XML. As a feature to detect clashes, Liquibase stores also a MD5 hash of the changeset source code itself, so that it is also able to detect that the content of the change "XY" itself was changed. In these case you must manually figure out what went wrong and teach your developer colleagues not to change any changeset after it has has been applied by any third party.

If you do not want to switch/introduce a new Database Version Control tool like liquibase I think you could easily extend your custom approach towards this direction. Simply maintain a list of applied changes and you'll be able to tackle branches well as long as your filenames don't match.

Upvotes: 2

Mark O'Connor
Mark O'Connor

Reputation: 78021

I use liquibase to manage my database migrations.

It uses an XML syntax to describe each "changeset" to the database. While this can be a barrier to adoption, I find it makes it much easier to merge contributions from different developers into the same file.

<changeSet id="bob-20130115-1" author="bob">
    <createTable tableName="commontable">
       ..
       ..
    </createTable>
</changeSet>

<changeSet id="tom-20130115-1" author="tom">
    <addColumn tableName="commontable">
        <column name="newcolumn" type="varchar(255)"/>
    </addColumn>
</changeSet>

A second advantage of using liquibase is that it supports rollback. In the event of conflict we can undo changesets, taking us back to the last stable release, fix the migration file and perform a fresh update.

If you don't have the luxury of switching tools, then I'd suggest perhaps adopting liquibase's approach of treating each migration during DEV as a "changeset". Use a single migration file for each iteration and add a comment to the start and end of each developer's contribution. In the event of merge conflicts, don't attempt to re-write the SQL, instead re-order so that one developer's change comes before or after the others. (The approach of forcing the other developer to create another file is valid, but I've found it a difficult to implement in practice, when a third developer comes along later and conflicts with the next number in the sequence....)

Managing the Integration environment without a rollback capability is tricky.... I don't know any easy method apart from rebuilding the database from scratch everytime. Perhaps the only effective way is to follow VonC's advice and keep the number of database changers to a minimum.

Notes:

  • Each developer using their own database is a given. You're using a migration tool, so there should be no objections. It simplifies issues, especially when you need to blow away and rebuild your schema. It also forces you to address the "test data" challenge.... Grinds my gears when I work on a project where the test data is located in a single database instance....
  • I've discovered recently that SQL Alchemy migrate has the same rollback capabilities as liquibase. It forces you to write both the roll-forward and roll-back scripts for each migration.

Upvotes: 4

VonC
VonC

Reputation: 1329582

two developers add a migration script at the same time

They need to do their modifications on their own branch, and trigger separate review on Gerrit.
An integrator (or one of the two developers) will be in charge to merge and review the two development efforts.

That, or the communication between the two needs to improve, in order for "dev 1" to not push anything without having check with the "dev 2" if "dev 1" needs to get some patch (representing "dev 2"'s work) first in "dev 1"'s working tree.

Upvotes: 0

Related Questions