CODEWITHSUNDEEP
go
Bohua Jia
Bohua Jia

Reputation: 77

Batching in golang

I want to implement my program to handle batching which can improve the efficiency of the process.

I confused if I set a function to wait for seconds and then process all the messages received from a client, how can I do that without interrupting the infinitive loop. e.g.

for{
    msg <- listenUDP
    batching(msg)
}

And also I am not sure if I make batching function can wait without interrupting the for loop, when a new 'msg' received and batching function still running. Will the system call a new batching function? If do so, how can I force the system to use the existing batching function rather than call a new one?

Upvotes: 0

Views: 929

Answers (1)

Sai Ravi Teja K
Sai Ravi Teja K

Reputation: 655

Based on the details you have given, one possible approach is that you can maintain an in-memory list of events yet to be processed and then invoke a separate goroutine to process each batch after a specified time or if the list reaches a specific size.

That way your infinite loop can continue to process messages while you process messages in batches. Based on the requirement you can have communication between the main goroutine and batch goroutines

It is dependent on the implementation actually. For example, you can spawn a worker thread and then forget about it i.e. the main goroutine just continues to receive messages. The pseudo-code for that could be like below:

for each event received:
    check if the time limit has crossed or if the message list has crossed
    if yes then spawn a new goroutine and forget about it
    if not push the message to list and continue to the next message

Upvotes: 1

Related Questions