CODEWITHSUNDEEP
c#optimizationio
Clayton Leung
Clayton Leung

Reputation: 269

Write file need to optimised for heavy traffic part 2

For anyone interest to see where I come from you can refer to part 1, but it is not necessary.

write file need to optimised for heavy traffic

Below is a snippet of code I have written to capture some financial tick data from the broker API. The code will run without error. I need to optimize the code, because in peak hours the zf_TickEvent method will be call more than 10000 times a second. I use a memorystream to hold the data until it reaches a certain size, then I output it into a text file.

The broker API is only single threaded. 

void zf_TickEvent(object sender, ZenFire.TickEventArgs e)
{

    outputString = string.Format("{0},{1},{2},{3},{4}\r\n",
                        e.TimeStamp.ToString(timeFmt),
                        e.Product.ToString(),
                        Enum.GetName(typeof(ZenFire.TickType), e.Type),
                        e.Price,
                        e.Volume);

    fillBuffer(outputString);

}

public class memoryStreamClass
{
    public static MemoryStream ms = new MemoryStream();
}

void fillBuffer(string outputString)
{

    byte[] outputByte = Encoding.ASCII.GetBytes(outputString);

    memoryStreamClass.ms.Write(outputByte, 0, outputByte.Length);

    if (memoryStreamClass.ms.Length > 8192)
    {
        emptyBuffer(memoryStreamClass.ms);
        memoryStreamClass.ms.SetLength(0);
        memoryStreamClass.ms.Position = 0;
    }
}

void emptyBuffer(MemoryStream ms)
{
    FileStream outStream = new FileStream("c:\\test.txt", FileMode.Append);

    ms.WriteTo(outStream);
    outStream.Flush();
    outStream.Close();
}

Question:

  1. Any suggestion to make this even faster? I will try to vary the buffer length but in terms of code structure, is this (almost) the fastest?

  2. When memorystream is filled up and I am emptying it to the file, what would happen to the new data coming in? Do I need to implement a second buffer to hold that data while I am emptying my first buffer? Or is c# smart enough to figure it out?

Thanks for any advice

Upvotes: 1

Views: 160

Answers (2)

Andrey Atapin
Andrey Atapin

Reputation: 7955

As for the second question: since your program is a singlethreaded one, incoming data will be ignored while the stream i sbeing flushed, because the execution flow will stop at flushing. But I suppose the framework you use works asynchronously. Here you may not avoid multithreading. You will have to synchronize access to your stream. And the best way is to implement Producer-Consumer pattern, as @usr have said.
Multithreading is an advanced topic, but it is a must know in modern programming. Consider to learn it and don't ignore.

Upvotes: 0

usr
usr

Reputation: 171216

The fastest way to do this is to have one (or more) threads put byte[]'s into a BlockingCollection and have one thread take the items out as fast as it possibly can and write them into a file. That way your producers and the file-writing consumer are totally decoupled. You will be able to sustain very high load doing that.

Upvotes: 2

Related Questions