Keeping Go microservices in different repositories on Github

Question

I'm working on a micro-services project. For this, I'd like to have one Go package per service, all included in a parent package for the project. It looks like this:

.
└── github.com
    └── username
        └── project
            ├── service1
            └── service2

I think this structure allows to comply with the Go conventions on package names and paths. A consequence of this is that all my microservices end on the same repository on Github, as the repository will be at depth 3 in the URL. I think this may become an issue in the future if the codebase becomes large. It may also add complexity for the CI/CD pipeline, for example a change to one service would trigger a build for all other services and the code to clone would be unnecessarily large.

Is there a way to avoid this conflict between the Go conventions and the way Github works ? Or this a problem that have to be solved during the CI/CD work ?

Answer 1

There is a good way to work with separate repositories and Go microservices: Go plugins. In short:

• Build a base image that implements shared functionality.
• Have that base image look for a Go plugin somewhere in the container
• In derived images, compile the functionality as a Go plugin and place it where the base image can find it.

For Go/gRPC, I have put a base image that does this here.

Answer 2

Versioning sub-packages that a Go project depends on can be track by git tagging. So using Go-modules one is encouraged to move sub-packages into their own git repos.

If the majority of your solution will be written in go, I'd suggest leveraging go modules.

This blog post explains how to manage Go-modules' go.mod with regard to package dependencies and their associated version git tags:

• https://blog.golang.org/migrating-to-go-modules

Answer 3

What you're talking about is popularly called "monorepo" these days. While I personally like having all my projects in the own independent repositories (including microservices and everything else), there are a number of proponents of having all code for a company in a single repository. Interestingly, both Google and Facebook use monorepos though it must be said they have built a lot of fancy tooling to make that work for them.

One important thing to note is that your repository is a separate thing from your architecture. There is not necessarily any correlation between them. You can have microservices all in a single repo and you can have a monolith divided up into several repos; the repository is only a tool to store and document your code base, nothing more.

While researching the topic, here are some of the advantages and disadvantages taken from a number of articles across the web:

Monorepo Advantages

• Ease of sharing modules between projects (even in microservices, there are often cross-cutting concerns)
• One single place to see and know what code exists - especially useful in large companies with lots of code
• Simplifies automated and manual code review processes
• Simplifies documentation rather than pulling from multiple, disconnected repos

Monorepo Disadvantages

• Massive codebase can be challenging/slow to check in/out to local
• Without very clear, strict guidelines it can be easy to cause tight coupling between products
• Requires (slightly) more complex CI/CD tooling to partial-release
• Depending on repository platform, very large codebases can affect performance

And here's a good discussion on the pros and cons of monorepos, and here's one specifically related to switching TO a monorepo with microservices architecture. Here's one more with lots of links both pro- and against-monorepo.

Like so many other things in programming and especially in SOA, the right solution for you depends on a number of factors that only you can determine. The main takeaway is that big and small companies have been successful with both options and many in between, so choose carefully but don't worry too much about it.