git submodules vs maven artifacts

Question

Our rest service development is handicapped and slowed down with current structure. I have tried to capture it in a picture below. Some explanation about the diagram

• prd-common, RestServiceA and RestServiceB are separate github repos
• v1 has two submodules. v1-lifecycle and v1-utils. v1-lifecycle depends on v1-utils. Similarly, v2 has same dependencies
• both v1 and v2 depend on common which is another maven module
• Both v1 and v2 are disjointed meaning they don’t know about each other
• RestServiceA needs code from v1-lifecycle and v2-lifecycle so it directly depends on them
• RestServiceB needs code only from v2-lifecycle so it directly depends on it

Most of the code is under v1 or v2. RestServiceA/B don’t have much code in them apart from resources, configurations etc. So, there is a tight coupling here.

Problem with this approach is that, for a given feature I make changes in let’s say v1, I need to build the jar, update its version in let’s say RestServiceA and test. This needs to repeat for every new commit. Additionally, I would need to publish my own snapshots to test outside of local.

To avoid this mess, I was thinking of using github submodules. However, my knowledge is limited. Can someone help if below understanding is correct and if I am missing any obvious loopholes if were to choose this approach. I’m open to any alternative approach. Thanks!

• If someone is making local changes to let’s say v1, they push the commit and do a github submodule update to get the latest hash in let’s say RestServiceA and test. This will be much faster than building entire jar and improve on time to market.
• At times, we need to test code from a developer in production for A/B testing. In those cases, RestServiceA can point to the developer’s special fork and branch temporarily, create a build, test and revert. However how can we avoid developers to not commit to .gitmodules in the main repo? Maybe write a unit test and fail it explicitly if the url/branch etc. changed
• We can still publish maven artifacts as part of releases, so if RestServiceB,RestServiceC want to stick to artifact and not submodule, they can do so.

Answer 1

We have a similar project structure and ended up with the following solution.

1. There's a Git repository per each module ("module repositories"). In your case that would be all repositories that you've mentioned in the post:

v1-lifecycle, v1-utils, v2-lifecycle, v2-utils, prd-common, RestServiceA, RestServiceB, RestServiceC

2. Each module repository contains an usual Maven project that can be built independently, this is more or less like what you have now, but...
3. ...The module repositories do not have submodules, they only refer to each other as Maven dependencies.
4. On every push to the module repositories CI uploads the new artifacts to Maven if the tests pass (or it doesn't if not).
5. In addition to module repositories, there's a single ("parent") Git repository that contains other module repositories inserted into its tree (I'll explain below what it means). So the parent repository contains all the code in some sense.
6. The parent repository's pom.xml includes all other modules in its trees using Maven's <modules> tag:

    <modules>
        <module>module1</module>
        ...
        <module>moduleN</module>
    </modules>

And the tree structure of the parent repository looks like:

├── module1
├── ...
├── moduleN
└── pom.xml

This allows us to build and test all the projects before committing and before publishing the artifacts using mvn test, mvn integration-test, or mvn package.

Also this architecture makes cross-module refactorings much easier as IDE opens all the code. So we do all the development from the parent repository directly.

I wrote an article describing this setup on a tiny example of just 2 modules.

Here comes full disclosure: the project we're developing is Git X-Modules (is a replacement to Git submodules) and it uses itself instead of Git submodules to insert module repositories into the tree of the parent repository.

So by inserting module repositories into the parent repository I mean using of either:

• Git submodules; OR
• Git X-Modules, if you choose that way like us

to create module1...moduleN directories at the root of the parent repository.

The difference is that in case of Git submodules the module1...moduleN directories at the root of the parent repository will be links to their corresponding Git repositories. E.g. if we change them from the parent repository, we have to commit and push into each of the individual module repository.

In case of Git X-Modules they all be just normal Git directories, so we have to commit and push only once, and on the server side the push is converted to pushes into individual module1...moduleN repositories. I hope you get the idea. The branching is also easier in this case.

Regarding your last question, afaiu this was needed because you don't want to test changes before committing them. When having the parent repository with Maven modules, you can just run "mvn test", "mvn integration-test", or even perform manual testing of all the changes before committing and pushing, so the question becomes irrelevant, as I understand.

In the worst case the developer can create a new branch directly in the parent repository and push just that branch to make CI test it and then merge the branch (or cherry-pick from it) it everything is ok. No temporary upload to Maven is involved.