I/O Errors writing to log file since I changed method of archiving it

Question

I wrote a couple of articles the other day about problems archiving a large log file > Out of memory error archiving a log file

And then the use of GC.Collect() which seemed to solve the issue > Should GC.Collect() be called regularly?

However as you can see from the comments people were against the use of the Garbage Collector so I rewrote my archive process to use a simple FileInfo.CopyTo(path); method.

However since doing this, although the log file is archived without problem I have noticed that I am now getting a lot of I/O exceptions that I never used to get when trying to write to the log file.

I have no idea if this is related to the change of archive method but it does seem a bit of a coincidence that as soon as I changed the method from reading in the old logfile as a string and writing it out to a new file, then creating a brand new log file for the day that I suddenly start getting exceptions.

The log file has always been big and busy with methods logging to it and the current level of debug is set to "LOW". I have HIGH, MEDIUM and LOW on all debug calls and then a global setting to which level I want to log to which is currently set to LOW, so only the important calls such as entry calls/return value to and from methods with the parameters and some main debugging I need are logged.

I can understand if two processes were both trying to write to the file at the same time that an exception would be thrown. But how come the actual details of the Exception are able to be written to the log file.

If it is locked because another process is trying to write to it then surely it would be locked to prevent the details of the exception out to the file as well?

I am just worried I am missing important logging and that it has suddenly started happening since the change of archive method.

Example

OpenBet - Exception in GetBetStatus; reRun: False; Exception: I/O Error writing to log file: C:\ProgramData\BrainiacV2\Logfile.log; The process cannot access the file 'C:\ProgramData\BrainiacV2\Logfile.log' because it is being used by another process.

StackTrace: at Api_ng_sample_code.HelperLib.LogMsg(String msg, String debugLevel) in c:\Users\rreid\Documents\Visual Studio\2012\Projects\BetfairBOT2\Api-ng-sample-code\HelperLib.cs:line 626 at Api_ng_sample_code.BetfairBOT.GetBetStatus(Int32 systemPK, Boolean reRun) in c:\Users\rreid\Documents\Visual Studio 2012\Projects\BetfairBOT2\Api-ng-sample-code\BetfairBOT.cs:line 4669;

The logging method just has this at the bottom of it once it has decided to log or not and the I/O exception is obviously being thrown.

try
{
    File.AppendAllText(this.LogFilePath, msg, Encoding.UTF8);
}
catch (IOException exception)
{
throw new IOException("I/O Error writing to log file: " + this.LogFilePath + "; " + exception.Message.ToString(), exception);
}
catch (Exception exception)
{
    throw new Exception("Error writing to log file: " + this.LogFilePath + "; " + exception.Message.ToString(), exception);
}

I don't understand why the message cannot be written out to the log file due to locking by another process but the actual exception error message is able to be written out.

There are lots of methods being called by timers so processes run against each other all the time and I use the database to do some basic locking to prevent two important methods running at the same time etc as I don't want to go down a multi threaded application route.

It just seems odd. Is there a way to queue messages in memory from the various timed jobs somehow so that they can be logged?

I was thinking of saving them all to the database then outputting them in date order every 10 minutes or so, so that only one process was writing to the file but the amount of database calls I would be making would be horrendous.

Is a better logging system possible in order to prevent I/O errors?