CODEWITHSUNDEEP
c#.netfile-io
Nick
Nick

Reputation: 1424

Efficient file read/write between two servers

Server 1 app (C# .NET) does file writes and Server 2 app (C# .NET) does file reads (of the same files written by Server 1 app). In this scenario:

  1. Which server should be the storage point for fastest, most efficient reads and writes? I suppose it should be Server 1, as it's writing the most. Slight latency in cross-server reads may be negligible.

  2. Which .NET API (i.e. FileStream, File.WriteAllText etc.) should be used for most reliable network file access (single-thread write but multi-thread read)?

Both servers are located within the same datacenter, both run Windows Server 2012 and have SSD storage.

Upvotes: 2

Views: 1314

Answers (1)

SashaDu
SashaDu

Reputation: 376

You may do a very simple test. For example, I created a small test app which writes and reads 1000 files with 1000 bytes each to local drive and to network drive. Here is the code:

    private void btnStart_Click(object sender, EventArgs e)
    {
        int count, size;
        count = Int32.Parse(tbCount.Text);
        size = Int32.Parse(tbSize.Text);
        string path = tbPath.Text;

        string stringToWrite = new string('A', size);

        System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
        sw.Start();

        for(int i=0;i<count;i++)
        {
            string fileName = System.IO.Path.Combine(path, i.ToString() + ".tst");
            System.IO.File.AppendAllText(fileName, stringToWrite);
        }

        sw.Stop();
        tbLog.Text += String.Format("{0} files with length {1} saved to {2} in {3}ms"+Environment.NewLine, count, size, path, sw.ElapsedMilliseconds);
    }

    private void btnRead_Click(object sender, EventArgs e)
    {
        int count;
        count = Int32.Parse(tbCount.Text);
        string path = tbPath.Text;

        System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
        sw.Start();

        for (int i = 0; i < count; i++)
        {
            string fileName = System.IO.Path.Combine(path, i.ToString() + ".tst");
            string temp = System.IO.File.ReadAllText(fileName);
        }

        sw.Stop();
        tbLog.Text += String.Format("{0} files loaded from {1} in {2}ms" + Environment.NewLine, count, path, sw.ElapsedMilliseconds);
    }

(First function writes files, second function reads files; size, count and path are defined from UI).

Here are the results:
  Writing 1000 files 1000 bytes each:
    Local drive: 800ms
    Network drive: 9000ms
  Reading 1000 files 1000 bytes each:
    Local drive: 500ms
    Network drive: 5000ms

So you see that somebody should pay for remote access - writer or reader. You see also that writing to remote location is twice longer than reading from remote location.

I think that the decision should be made based on the logic of your application and server functions - which server is more busy, which server should process the information fast? For example, if the first server gets data in a very high rate and should save all the data quickly to the files in real time, and the second server may queue the file processing and take its time without any rush - then the files should be saved locally on the first server. If the first server just writes files and its processor/IO is not too busy but the second server is loaded to its maximum by processing the files - let the first server to deliver the files to the second server's local drive.

Think also about scalability - what if in the future you will need 5 servers of type "Server 1" writing files for the single server 2? Or, 5 servers of type "Server 2" concurrently processing files written by the single server 1? Or many servers writing and reading files concurrently? May be the database or some queue solution would be better if these options are relevant.

But, if you does not care about all these things and just wish to have better overall performance - then, again, writing files locally and reading from remote location is faster.

Upvotes: 1

Related Questions