CODEWITHSUNDEEP
architectureazure-web-app-serviceiotazure-eventhubazure-iot-hub
Muhammad Ummar
Muhammad Ummar

Reputation: 3631

Azure Solution for Remote monitoring

I have build a test application for some demo purposes which is based on IoT and Industry 4.0 concept. Following are important points.

  1. I have build a background application on Windows IoT (on Raspberry Pi 3) to collect sensors data and push to MS Azure cloud. There could be many simultaneous IoT devices.
  2. On Azure side IoT Hub receives device to cloud messages.
  3. Then Data is passed to stream analytics job which passes on the data to Event hub.
  4. Event hub has 2 consumer groups in my configuration, which both are used by Azure web app.
  5. Default Consumer group is used by WebJob to store historical data, while Custom consumer group named as Consumer group 1 in image is used by MVC 5 app.
  6. MVC 5 app uses SignalR to pass on data to connected user to update dashboard.
  7. For demo purpose this scenario seems to be working fine, where we have 5 or 10 simultaneous connection to cloud webapp.

Following is the Architecture diagram of setup

IoT solution Architecture diagram

Problem

Now for real scenarios there could be many simultaneous users to cloud application and there is the point when cloud comes into play, and it can horizontally scale-up the webapp to many instances, It could create problem while directly reading events from Event Hub in that case and not all web-app instances will receive all events Which is wrong, and I think this is the reason in MS Azure remote monitoring Architecture diagram, that webapp is not reading messages from Event Hub directly.

Question

What could be best/qucik changes to above architecture to resolve the problem and still show realtime events to clients?

Upvotes: 1

Views: 679

Answers (1)

Rita Han
Rita Han

Reputation: 9720

Reading event data directly from event hub is limited by these quotas and limits.

Maximum number of partitions per Event Hub is 32.

Note that a single partition within a consumer group cannot have more than 5 concurrent readers connected at any time.

So, for example, if you have one event hub, there are 32*5 concurrent readers supported at most.

If you have many more simultaneous users(>32*5) you need increase the number of event hub.

It could create problem while directly reading events from Event Hub in that case and not all web-app instances will receive all events...

There some possible reasons:

  1. If one or more readers disconnect due to Epoch setting.
  2. If you don't specify a partition key when publishing an event, a round robin assignment is used. Then receive from a specified partition will lose other partitions' event data.

I test the scenario that five simultaneous readers receive event from one partition. No events loss.

Hope this is helpful for you. If there is any concern, please feel free to let me know.

Upvotes: 1

Related Questions