Speed up msbuild with fsutil hardlink create instead of slow copy tasks

Question

I'm trying to do speed up our build (csharp, msbuild, .net 3.5). Replace copy with fsutil hardlink create.

Previously, I almost got it with running a script over sln files and making the dll references private = false, then having a post build event create the hardlinks. Problem is transitive dependencies are not included. So, I think I need to reference the ResolveAssemblyReference task in msbuild to get the transitive dependencies I need to hardlink.

Any ideas?

This person tried the same thing, but did not post an final solution.

To be clear: What I want is to keep separate bin directories, but instead of copying file from one to another to create a hard link from a source (of a reference or a dependency) to the destination (the current project's bin). Which is much faster, and gives approximately the same effect as copying.

jawspeak · Accepted Answer

This is supported in VS 2010. But not 2008. See the UseHardLinksIfPossible option to Copy in _CopyFilesMarkedCopyLocal.

See also http://social.msdn.microsoft.com/Forums/en/tfsbuild/thread/9382a3d8-4632-4826-ad15-d5e845080981, http://msdn.microsoft.com/en-us/library/ms171466(v=VS.90).aspx for context.

Override the _CopyFilesMarkedCopyLocal target. We add something like this to the csproj files at the bottom: (This is auto added to the file with a nant task on every full nant build, so every project benefits).

Our new targets file is:

See the msbuild 4 Copy UseHardlinksIfPossible option. I backported that to 3.5 through decompiling and reimplementing. The relevant logic in CopyFileWithLogging was:

      // The port from 4.0's task that allows hardlinking
        bool hardlinkSucceeded = false;
        if (UseHardlinksIfPossible)
        {
            if (File.Exists(destinationFile))
            {
                FileUtilities.DeleteNoThrow(destinationFile);
            }
            if (!TwNativeMethods.CreateHardLink(destinationFile, sourceFile, IntPtr.Zero))
            {
                var win32Exception = new Win32Exception(Marshal.GetLastWin32Error());
                Log.LogMessage(MessageImportance.High, "Hardlinking had a problem {0}, will retry copying. {1}", new object[] {win32Exception.Message, win32Exception});
            }
            hardlinkSucceeded = true;
        }
        if (!hardlinkSucceeded)
        {
            Log.LogMessageFromResources(MessageImportance.Normal, "Copy.FileComment", new object[] { sourceFile, destinationFile });
            Log.LogMessageFromResources(MessageImportance.Low, "Shared.ExecCommand", new object[0]);
            Log.LogCommandLine(MessageImportance.Low, "copy /y "" + sourceFile + "" "" + destinationFile + """);
            File.Copy(sourceFile, destinationFile, true);
        }
        // end port

Also had to add these:

// decompiled from 4.0
internal static class TwNativeMethods
{
    [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)]
    internal static extern bool CreateHardLink(string newFileName, string exitingFileName, IntPtr securityAttributes);
}

// decompiled from 4.0
internal static class FileUtilities
{
    internal static void DeleteNoThrow(string path)
    {
        try
        {
            File.Delete(path);
        }
        catch (Exception exception)
        {
            if (ExceptionHandling.NotExpectedException(exception))
            {
                throw;
            }
        }
    }
}

// decompiled from 4.0
internal static class ExceptionHandling
{
    // Methods
    internal static bool IsCriticalException(Exception e)
    {
        return (((e is StackOverflowException) || (e is OutOfMemoryException)) || ((e is ExecutionEngineException) || (e is AccessViolationException)));
    }

    internal static bool NotExpectedException(Exception e)
    {
        return (((!(e is UnauthorizedAccessException) && !(e is ArgumentNullException)) && (!(e is PathTooLongException) && !(e is DirectoryNotFoundException))) && ((!(e is NotSupportedException) && !(e is ArgumentException)) && (!(e is SecurityException) && !(e is IOException))));
    }

    internal static bool NotExpectedReflectionException(Exception e)
    {
        return ((((!(e is TypeLoadException) && !(e is MethodAccessException)) && (!(e is MissingMethodException) && !(e is MemberAccessException))) && ((!(e is BadImageFormatException) && !(e is ReflectionTypeLoadException)) && (!(e is CustomAttributeFormatException) && !(e is TargetParameterCountException)))) && (((!(e is InvalidCastException) && !(e is AmbiguousMatchException)) && (!(e is InvalidFilterCriteriaException) && !(e is TargetException))) && (!(e is MissingFieldException) && NotExpectedException(e))));
    }
}

Speed up msbuild with fsutil hardlink create instead of slow copy tasks

Answers (2)

Related Questions