How to manage releases to minimise integration issues?

Question

We have many developers working on a project in Git. Right now we develop in separate branches and testing is done on these branches. The problem is when we integrate these branches back to master some new issues creep in. Reasons are:

1. Merge and conflict resolution issues.
2. This sounds far off but happens too often that a bug uncovered in one branch could have been caught by the test coverage done in another branch. So it could have been avoided.
3. In our setup the master branch is always available to users for beta testing. Since we integrate back to master quite late in release cycle we do not get sufficient amount of beta testing coverage.

Lately what we have tried:

As soon as a new branch is merged in master we merge it back to individual branches trying to keep the branches as close to master as possible. This minimises but does not solve the issue because of these two reasons -

1. Its possible that a branch that has caused an issue was actually merged on the last release day or later in the cycle.

2. The buggy branch since it was merged quite late in the cycle also does not get any coverage in beta testing (master is the beta testing branch).

What best practice do we follow to minimise this problem? I have this solution in mind -

1. We create a next "release" branch at the beginning of release cycle. So we create a release branch on Monday when Friday is the release date.
2. We complete testing in individual branches and merge all to be released changes in this "release" branch.
3. We give this "release" branch for beta testing as well as continue our integration testings on this branch.
4. If everything is fine we merge back from this branch to master. So we maintain one release branch and master.

This delays release by the number of days we decide for integration testing but appears to be more disciplined. Since this problem is not new, please give your suggestions or point me to the right ones on the best practices around this.

Answer 1

You could try a completely different approach - adopting a more Continuous Delivery approach means that everyone works on master/trunk, and you utilise Feature Toggles to hide in-progress work from the end users.

This way, you only have small daily merges, rather than big ones (improving point 1), and you also more-or-less solve points 2 and 3.

Answer 2

Here is how I would go about doing it:

1. Create a Project branch. This is now the master as far as the project is concerned.
2. Each developer creates their own branch from the Project branch to work on.

3. Test all work done in the developer branch before any merging back to the Project. Merging to the project branch is in two steps:

   Step 1 REBASE the developer branch. That is merge the Project branch to the developer branch. Fix any problems and test to ensure no faults have crept in.

   Note: The extra testing here helps to pinpoint new problems (introduced through the merge) caused by integrating work from other developers into the current developers branch.

   Step 2 If all is OK, merge the developer branch back to the Project branch.

   Note: No testing is required here because the Project branch and the Developer branches are identical.

4. Once all the developers have completed their work for this release cycle and have successfully REBASED and merged to the project branch; test the project branch to ensure all is OK (regression tests).
5. Deliver the product for this release cycle. Delivery is done by REBASING the project branch, i.e. merging the master to the Project, fixing up any problems, testing (again!).

I know this sounds like a lot of extra work, and it also looks like some one has to administer the repository but if you keep at it, it becomes routine and the administration activity falls off. The testing activities also become more routine in that each developer learns the kind of testing required at each step and plans for it.

Good luck with your project! J. Isa

Answer 3

1) Merge conflicts

If at all possible, work on separate files. If it's not, you'll likely have to deal with conflicts. There are some tricks to dealing with them that some devs don't know. For example, say you get a merge conflict on file A. Let's say that you want your version of file A, and not master's (this is very common). Then, you can say git checkout HEAD -- /path/to/fileA. Then add/commit. This will make git use your changed file as-is, and not master's. Conversely, if you want master's version, you can do git checkout master -- /path/to/fileA, assuming your local master is up-to-date. Or you can just pick out any file from any branch with this method: it's pretty general and is really convenient at times.

2) In terms of "merge bugs", here's how we handle it:

We use Jenkins/BitBucket as our continuous integration tools, and they seem to work well together. We have a similar development model to yours, we branch right off master and then merge features right back in. However, we don't get these "bugs" from merging, because we force Jenkins to "fake the merge to master", and then run all unit tests on the "fake merge to master" before we're allowed to even merge the pull request into master. This really beats down integration bugs. If you don't have the tools to do this, you have to be really arduous to merge in master and run all your tests before merging. This is a pain in the butt: I suggest making Jenkins or some other automated CI tool do the job for you.

3) Release branches

These can be a good practice. We don't implement them, however. You may find that you turn into a "git secretary" rather quickly by implementing these. We've never actually tried them, so it's hard for me to say whether they're a good idea, but that's the argument I've heard against them from people who have tried them (hearsay, I know).

HTH. Good luck with your release(s).