Real time data integration how does Kafka, Hadoop, Avro and HDFS fit together and what kind of architectures is there for data integration

Question

I am trying to understand what architectures there is for real time data integration and how all the pieces fit together.

I have tried to research on the internet but I could not find good resources.

Can anyone explain how to use for example hadoop kafka and other components to make a real time data integration.

And if possible can you provide where real time data integration is used. Any use-case.

If you know any resources/ tutorials I would be more than happy if you share them

Thanks in advance

Answer 1

If I may humbly offer up two talks I have done on this.

• Conceptually, what & why we can about real time integration: https://talks.rmoff.net/said0W/the-changing-face-of-etl-event-driven-architectures-for-data-engineers

• In practice: https://talks.rmoff.net/pZC6Za/apache-kafka-and-ksql-in-action-lets-build-a-streaming-data-pipeline