CODEWITHSUNDEEP
c#streamwriter
Oleg Vazhnev
Oleg Vazhnev

Reputation: 24067

how to store information on disk frequently without introducing delays?

I really don't want to introduce any delays in my high frequency trading software and at the same time I need to store thousands of lines of logs every second. 1 ms delay would be huge, I only agree to have 0.01-0.05 ms delay.

*Now*I just allocate 500 Mb in memory at start-up, store logs there and when application finish I put this log on disk.

However now I realized that I want more logs and I want them during application execution. So I now want to store logs during application execution (probably once per minute or once per 10 minute). How slow StreamWriter.WriteLine is? Would it be slower than just "adding to preallocated collection"?

Should I use StreamWriter.WriteLine directly (is it syncrhonous or asynchronous, is AutoFlush option affects perfomance?). I also can use BlockingCollection to add items to log and then use dedicated thread to process this blocking collection and to store logs on disk in another thread.

Upvotes: 0

Views: 135

Answers (3)

oleksii
oleksii

Reputation: 35905

Don't

  • Reinvent a wheel

Do

  • Use a logging framework
  • Properly configure loggers and levels for each logger
  • Use sync logging for memory (it's simple and fast, but has problems with event persistence onto drive) and async for IO (it is difficult to get right, slow, harder to test) loggers

If you hadn't done so, check out log4net and NLog, this will be a good place to start.

Upvotes: 1

faester
faester

Reputation: 15076

Use log4net as Andre Calil suggests. It logs to SQL, disks and whatnot and is extremely customizable. It can seem a bit complicated at first, but it is worth the effort.

What you need is probably the RollingFileAppender. log4net is in nuget, but you should read the documentation at the log4net site. Start by looking at the appender config.

Upvotes: 0

René Kolařík
René Kolařík

Reputation: 1286

Probably you could store your logs in circular buffer and spawn a new thread of execution which will just send data from that buffer in shared memory to disk.

Upvotes: 0

Related Questions