Merge strategy when files moved between submodules

Question

We are working on a big Unity3D project and dependencies were managed through submodules.

Because that introduce no restriction between the submodules when it comes to dependency management, the submodules become interconnected over time and code gets scattered across several submodules, instead of preserving a single responsibility for each submodule.

I had to resolve dependencies in multiple repositories and shift classes between repositories.

Now, the submodules are independent, but I need to merge/rebase the latest change from a college.

But because his submodules now contain different files than my submodules, the files moved to another repository gets added again and the changes of files moved away can't be integrated, because these files resides on his parent-repo in another submodule than on my parent-repo.

I know this is pretty worse, but what's the best strategy to take to integrate all changes in every submodule without missing something or adding the files again?

The only thing I can think of is filtering for the files I moved, and move the files on his side first before merging.