CODEWITHSUNDEEP
azureazure-stream-analyticsazure-eventhub
saran k
saran k

Reputation: 327

How can I handle multiple sources of data with single event hub name space

I'm getting data from 20-30 API's and pushing data to the event hub, I need to process data from the event hub at Azure stream analytics in different ways, so how can I identify data from different API's or sources with an azure stream analytics. I send the data like below in my code:

module.exports = async function (context, myTimer) {
    const {  EventHubClient, EventData, EventPosition, OnMessage, OnError, MessagingError } = require("@azure/event-hubs");
    const connectionString = 'Endpoint=Connection string'
    var axios = require('axios')
    context.log('ssss')
    const client = EventHubClient.createFromConnectionString(connectionString);
    context.log('sssaaa')
    var response = await axios.get('https://nseindia.com/live_market/dynaContent/live_watch/stock_watch/nifty500StockWatch.json')
    // var sendBatchData = [
    //     {body:{response['data']['data']}}
    //     ]
    const datas = [
        { body: { "message": "Hello World 1" }, applicationProperties: { id: 1 }, partitionKey: "pk786" },
        { body: { "message": "Hello World 2" } },
        { body: { "message": "Hello World 3" } }
        ];
    // context.log(sendBatchData)
    await client.sendBatch(datas)
    context.log('message sent')
    // async function main():Promise<void> {


    // }

    // var timeStamp = new Date().toISOString();

    // if (myTimer.IsPastDue)
    // {
    //     context.log('JavaScript is running late!');
    // }
    // context.log('JavaScript timer trigger function ran!', timeStamp);   
};

But on the azure string analytics site, I'm getting the data like belowe:

{
    "message": "Hello World 3",
    "EventProcessedUtcTime": "2019-09-23T09:52:09.1421367Z",
    "PartitionId": 0,
    "EventEnqueuedUtcTime": "2019-09-23T09:51:38.8270000Z"
  },
  {
    "message": "Hello World 1",
    "EventProcessedUtcTime": "2019-09-23T09:52:08.6420937Z",
    "PartitionId": 0,
    "EventEnqueuedUtcTime": "2019-09-23T09:51:38.8270000Z"
  },
  {
    "message": "Hello World 2",
    "EventProcessedUtcTime": "2019-09-23T09:52:09.1421367Z",
    "PartitionId": 0,
    "EventEnqueuedUtcTime": "2019-09-23T09:51:38.8270000Z"
  }
]

I'm not seeing anything replated to my partition key. My Objective is that when I'm sending data from multiple Api sources how can I differentiate at Azure stream analytics

Thanks in advance

Upvotes: 1

Views: 1056

Answers (1)

Jay Gong
Jay Gong

Reputation: 23782

Per my knowledge, ASA event hub stream input can't make a distinction between the data from different source APIs automatically.

Two ways for you reference:

1.When your APIs pushing data to the event hub,please specify a value as PartitionId. Then you could query the partition id in the ASA query:

enter image description here

2.Add custom unified column for all APIs. Such as: SourceFlag: API1,API2,API3...,then you could distinguish the data based on this custom column in the ASA query.

Upvotes: 1

Related Questions