Compile XPC service with a different architecture than the client

Question

I'm working on a macOS application that deals with a few external dependencies that can only be compiled for intel (x86_64) but I want the app to run natively on both arm and x86_64.

One idea I have been playing with is to move the x86_64 dependencies to an xpc service compiled only as x86_64 and use the service only the intel machine. However, I can't figure out how to setup my project to compile everything at once...

Any ideas? Is this even possible? If not, I'm open to suggestions...

Thanks