CODEWITHSUNDEEP
cposixmemory-managementmessage-queue
Crazy Chenz
Crazy Chenz

Reputation: 13208

How do you conserve memory when receiving messages from POSIX queues?

How do you conserve memory when receiving messages from POSIX queues?

It seems that when using POSIX queues in a multiprocess / multithreaded environment, there is no thread safe way to dequeue a message into a buffer that is anything smaller than the max_msgsize.

Are there any standard solutions to this problem? Or is it even a problem?

I am well aware that there are other really great libraries to do this, but I just wanted to include a completely standard solution for users if they don't want to deal with dependencies.

FYI, I am trying to queue up potentially hundreds of megabytes per message and have a pool of processes with multiple threads each dequeuing the messages for processing.

Thanks, Chenz

Upvotes: 1

Views: 237

Answers (2)

frankc
frankc

Reputation: 11483

I think there is no really good way to do this. Here is an idea, but I think you will find it performs badly due to lock contention:

Have one static buffer that is equal the size of the maximum message. Because there is a shared buffer, now your dequeue process must look like this:

  • lock sempaphore
  • dequeue into static buffer
  • figure out the real size of the message
  • copy from static buffer to a thread local buffer that is the actual size of the message
  • unlock semaphore

There is the overhead of having the static buffer, but depenidng on the distribution of the sizes of your messages, you are still likely to see a reduction in total memory usage. However, now you have to deal with the contention for the static buffer, which is likely to be great especially when a few large message arrive in a row. If very large messages are rare, then this might not be a terrible solution.

Upvotes: 0

sdg
sdg

Reputation: 4733

The POSIX queue interface, as you note, does not allow you to query the size of a message.

In effect, therefore, all messages may be at the maximum size as configured by the queue definition, and you have to assume that a simplistic implementation might well make use of that for ease of record-keeping.

Given that you are dealing with multi-megabyte messages as yo say, putting those messages into the queue is unlikely to be a good solution (unfortunately).

If your message rate is low (to some definition of low) and you actually do have a reasonable upper bound, then just go ahead and try it out.

Barring that, your next best bet would be to use the queue as a work-order queue, and not as a work-item queue. Your work items would have to be stored differently, in files perhaps. Then in the queue you have a nice short filename, pointing to the location of the work-item to take care of.

Good Luck

Upvotes: 2

Related Questions