GitOps - Config in same repo or seperate repo?

Question

Firstly this is in the context of a mono-repo application that runs inside Kubernetes.

In GitOps my understanding is everything is declarative and written in configuration files such as YAML. This allows a full change history within git. Branches represent environments. For example:

• BRANCH develop Could be a deployed QA or staging environment
• BRANCH feature/foo-bar Could be a deployed feature branch for assessment
• TAG v1.2.0 Could be latest version running in production This makes sense to me, any and all branches can be deployed as a running version of the application.

QUESTION I remember reading ... somewhere ... configuration should live outside of the main repository, inside another "configuration repository". From memory the idea is an application should not know a particular configuration... only how to use a configuration?

Is this true? Should I have an application repo and an application configuration repo? e.g.

• app repo: foo-organisation/bar-application
• config repo: foo-organisation/bar-application-config

Where the branching model of config for different environments lives inside that repository? Why and what are the advantages?

Otherwise should it just live inside a directory of the app repo?

Answer 1

ArgoCD has a best practices page saying that it is best to have a separate repo for your Kubernetes manifest

It provides a clean separation of application code vs. application config. There will be times when you wish to modify just the manifests without triggering an entire CI build. For example, you likely do not want to trigger a build if you simply wish to bump the number of replicas in a Deployment spec.

Cleaner audit log. For auditing purposes, a repo which only holds configuration will have a much cleaner Git history of what changes were made, without the noise coming from check-ins due to normal development activity.

Your application may be comprised of services built from multiple Git repositories, but is deployed as a single unit. Oftentimes, microservices applications are comprised of services with different versioning schemes, and release cycles (e.g. ELK, Kafka + ZooKeeper). It may not make sense to store the manifests in one of the source code repositories of a single component.

Separation of access. The developers who are developing the application, may not necessarily be the same people who can/should push to production environments, either intentionally or unintentionally. By having separate repos, commit access can be given to the source code repo, and not the application config repo.

If you are automating your CI pipeline, pushing manifest changes to the same Git repository can trigger an infinite loop of build jobs and Git commit triggers. Having a separate repo to push config changes to, prevents this from happening.

More info https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/

Answer 2

There's no real hard rule other than everything about your environment should be represented in a git repo (e.g. stop storing config in Jenkins env variables Steve!). Where the config lives is more of an implementation detail.

If your app is managed as a mono repo then why not use the same setup for the deployment? It's usually best to fit in with your existing release processes rather than creating something new.

One catch is that a single version of the app will normally need to support multiple deployments and the deployment config can often change after an app has been built and released. So there needs to be a 1 "app" to many "config" relationship. Whether that is implemented with files, directories, branches, or repos. They can all work, as long as they are versioned/released.

Another release consideration is application builds. For small deployment updates you don't want to build and release a complete new application image. It's preferable to just apply the config and reuse the existing artefacts. This is often a driver for the separate deploy/config repo, so concerns are completely separated.

Security can play a part. You may have system information in the deployment config that all developers don't need access to or you don't want to even have the chance of it making it into a container image. This also drives the use of separate repos.

Size of infrastructure is another. If I am managing multiple apps, the generic deployment config will not be stored in a single apps repo.

Answer 3

I strongly believe the configuration should live and be versioned together with the code mainly because changing it impacts the behavior of the system and therefore must follow the SDLC to ensure the change is not breaking anything else.

Let me enumerate some ideas:

• Credentials or secrets don't belong together with app configurations, they should be managed securely, they must be encrypted and I would recommend using something like Hashicrop Vault for them and obtaining them on runtime.
• Platform resources like database tables or Kafka topics should be part of the application, they seldomly change, and when they do I'd prefer to follow the SDLC to ensure they are not breaking the app.
• Infrastructure resources like CPU, Memory should be part of the application release, they could have a big impact on the behavior of the system and you don't want to experiment with them in higher environments.
• Endpoints to other services should be resolved using DNS, pointing to a different place should be a versioned change and should be tested to keep environment parity

If you keep the configuration in the same repository as the code, you'll follow a known process for anything that could cause an impact, not doing so opens the possibilities for human error, misconfiguration, untested changes reaching PROD, and many more failure scenarios.