CODEWITHSUNDEEP
azureazureservicebusservicebus
Lonefish
Lonefish

Reputation: 677

How to efficiently use Servicebus as communication through multiple webjobs

This is more of an architectural question which is probably an edge case of being allowed on SO. If it's not allowed, please any advise on how to update my question to be allowed

We have 7-8 Webjobs running that are supposed to process the same info in a specific order. What is the preferred way of doing this with servicebus? Is there best practice to use the servicebus for this?

At the moment I'm at something like this: enter image description here

See EDIT, this has been changed The packet comes in from our clients, goes through 2 jobs, and then gets sent to 3 jobs (not all of them are always necessary, they're sharing jobs. Some packets go to facebook, some to FTP, some both,.. And only after all 3 of them are done the last Job can proceed, since it's based on the info the 3 jobs give.

We could write states to the database, but we're specfically switching to servicebus to work transactional with our packets, have an easy way to recover on crashes and so on.

Also, would the best way be 4 different servicebusses or 1 bus where (if that is even possible) we requeue the same packet with a state field? For example add a statefield "Job1=OK" and Job 2 has a subscription on that field, so that it gets the packet when it's passed Job1. That would be ideal, it would also miminize the traffic to the servicebus since the packet stays on it until the end..

After some searching I found that I can forward messages, but can I forward them to the same topic too? And can I add data to them? And if I forward a message, does that count as an extra message in pricing?

EDIT: Updated the flow to this:

enter image description here

So there is no need to wait for the 3 jobs anymore. One Job just didn't need any waiting apparently, and the other 2 are merged.

Upvotes: 0

Views: 112

Answers (1)

Vova Bilyachat
Vova Bilyachat

Reputation: 19504

I would do it a bit different.

  • Why not to use different queues? For instance Step1, Step2, Step3 and so on (For sure you should do better naming). Then in web job you can create handlers

    public static void ProcessJob1([QueueTrigger("Job1")] CloudQueueMessage data, CancellationToken token){
   var message = data.AsString;
   //process event
   //Send event for next job2
}

  public static void ProcessJob2([QueueTrigger("Job2")] CloudQueueMessage data, CancellationToken token){
   var message = data.AsString;
   //process event
   //Send event for next job3
}

.....

  • I am using storage queue so i am sending json in service bus as soon as i recall you can ask it to deserialize

  • Merge webjobs to one webjob and use scaling, this means that you will use resources properly, because some events could use more resources some less
  • No need to redirect if you finish step 2 send one more event to other queue so web job will pick it up as new item.

In addition you can check other approach and this link but it will require to rewrite you app a bit

Upvotes: 1

Related Questions