CODEWITHSUNDEEP
javamultithreadingloggingasynchronouslocking
steve
steve

Reputation: 21

how to log asynchronously in a heavily multithreaded environment?

I am trying to to log asynchronously in a heavily multi-threaded environment in java on linux platform.what would be a suitable data structure(lock-free) to bring in low thread contention? I need to log GBs of messages. I need to do it in async/lock-free manner so I don't kill performance on the main logic(the code that invokes the logger apis).

Upvotes: 2

Views: 905

Answers (5)

rdalmeida
rdalmeida

Reputation: 1853

It is impossible to avoid queue contention as your logging thread will most likely log faster than your writer (disk i/o) thread can keep up, but with some smart wait strategies and thread pinning you can minimize latency and maximize throughput.

Take a look on CoralLog developed by Coral Blocks (with which I am affiliated) which uses a lock-free queue and can log a 64-byte message in 52 nanoseconds on average. It is capable of writing more than 5 million messages per second.

Upvotes: 0

Alexei Kaigorodov
Alexei Kaigorodov

Reputation: 13525

To reduce contention, you can first put log messages in a buffer, private to each thread. When the buffer is full, put it in a queue handled by a separate log thread, which then merges messages from different threads and writes them to a file. Note, you need that separate thread in any case, in order not to slowdown the working threads when the next buffer is to be written on disk.

Upvotes: 0

Peter Lawrey
Peter Lawrey

Reputation: 533530

I would use Java Chronicle, mostly because I wrote it but I suggest it here because you can write lock free and garbage free logging with a minimum of OS calls. This requires one log per thread, but you will have kept these to a minimum already I assume.

I have used this library to write 1 GB/second from a two threads. You may find having more threads will not help as much as you think if logging is a bottle neck for you.

BTW: You have given me an idea of how the log can be updated from multiple threads/processes, but it will take a while to implement and test.

Upvotes: 1

pauljwilliams
pauljwilliams

Reputation: 19225

Logback has an AsyncAppender that might meet your needs.

Upvotes: 3

Tuto
Tuto

Reputation: 92

The simplest way to do it is to write into multiple files - one for each thread.

Make sure you put timestamps at the start of each record, so it is easier to merge them into a single log file.

example unix command:

cat *.log | sort | less

But for a better / more useful answer you do need to clarify your question by adding a lot more detail.

Upvotes: 1

Related Questions