Event sourcing and microservices, messaging system

Question

We are designing our new system which will be written most likely from scratch, as the old one is really really old. It's really important for our system to keep audit trail logs of everything that happened in the system.

Due to the audit trail importance we decided to follow the event sourcing architecture for all of its benefits. Another key factor is that we have multiple teams who work on different "domains". Saying that, we would like to split each domain into it's own service (microservices architecture) so every team can work independently.

The biggest question that we have is who's going to be responsible for the event sharing between the microservices. For example, an event that happened in one of the services, should trigger another event in 2 other services as well. We can see 2 options:

• A central streaming service (AWS Kinesis most likely) with all services registering a publisher and a subscriber. Every service will publish all its events there and the consumers will decide if they "care" about the specific event and then do something or not.
• Multiple categorised streams (AWS Kinesis most likely) with specific service publishers and subscribers that really need to do something with the events that are published.

We've spent a lot of hours already trying to figure out what is the best option for us or what questions we should ask ourselves to take the right decision. I'm leaning more to the first option although i do have some concerns regarding performance and bottleneck for the consumers, since every service will listen to all events and then they will decide if they have to act on it or not.

Do you see any no-goes in the first option or the second one? Are we looking in the right direction or should we take a step back?

An image that describes a bit what we need to achieve (except CQRS) in a short scale is this one which is taken from here

Answer 1

I would tend to have a stream of events per publishing service. Then you have some level of categorization and don't have to deal (or at least not deal as much) with synchronizing consumption of multiple streams when you want to process events in order of emission.

The events will often have a natural key and you also often only need strict ordering after grouping by that key; accordingly, you can partition/shard the events.

There's nothing intrinsically preventing you from having both a firehose stream of all the events emitted by a publishing service and categorized streams of events so consumers can decide to not subscribe to streams they're not interested in.

Answer 2

Have you done a simple pros & cons table and compared them side-by-side? If you have a clear idea as to what your solution priorities are then the answer should be pretty obvious at the end of that process. If it's not clear then perhaps you're still trying to understand your solution priorities and which are the most important.

Note, by 'solution priorities' I mean system quality attributes like performance, maintainability, testability, and so on - but also taking into account architectural principles you might have, functional priorities of the product owner, commercial drivers, etc.

To more directly answer the OP, I'd start with a single event stream: you have everything in one place, which is simpler: easier to set-up and manage, debug, monitor, etc. Splitting things up just adds complexity. Sure, feel free to split it - but only if there's a clear reason to do so.