CODEWITHSUNDEEP
c#.netentity-framework
Stajs
Stajs

Reputation: 1370

Why is Entity Framework significantly slower when running in a different AppDomain?

We have a Windows service that loads a bunch of plugins (assemblies) in to their own AppDomain. Each plugin is aligned to a "service boundary" in the SOA sense, and so is responsible for accessing its own database. We have noticed that EF is 3 to 5 times slower when in a separate AppDomain.

I know that the first time EF creates a DbContext and hits the database, it has to do some setup work which has to be repeated per AppDomain (i.e. not cached across AppDomains). Considering that the EF code is entirely self-contained to the plugin (and hence self-contained to the AppDomain), I would have expected the timings to be comparable to the timings from the parent AppDomain. Why are they different?

Have tried targeting both .NET 4/EF 4.4 and .NET 4.5/EF 5.

Sample code

EF.csproj

Program.cs

class Program
{
    static void Main(string[] args)
    {
        var watch = Stopwatch.StartNew();
        var context = new Plugin.MyContext();
        watch.Stop();
        Console.WriteLine("outside plugin - new MyContext() : " + watch.ElapsedMilliseconds);

        watch = Stopwatch.StartNew();
        var posts = context.Posts.FirstOrDefault();
        watch.Stop();
        Console.WriteLine("outside plugin - FirstOrDefault(): " + watch.ElapsedMilliseconds);

        var pluginDll = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF.Plugin\bin\Debug\EF.Plugin.dll");
        var domain = AppDomain.CreateDomain("other");
        var plugin = (IPlugin) domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");

        plugin.FirstPost();

        Console.ReadLine();
    }
}

EF.Interfaces.csproj

IPlugin.cs

public interface IPlugin
{
    void FirstPost();
}

EF.Plugin.csproj

MyContext.cs

public class MyContext : DbContext
{
    public IDbSet<Post> Posts { get; set; }
}

Post.cs

public class Post
{
    public int Id { get; set; }
}

SamplePlugin.cs

public class SamplePlugin : MarshalByRefObject, IPlugin
{
    public void FirstPost()
    {
        var watch = Stopwatch.StartNew();
        var context = new MyContext();
        watch.Stop();
        Console.WriteLine(" inside plugin - new MyContext() : " + watch.ElapsedMilliseconds);

        watch = Stopwatch.StartNew();
        var posts = context.Posts.FirstOrDefault();
        watch.Stop();
        Console.WriteLine(" inside plugin - FirstOrDefault(): " + watch.ElapsedMilliseconds);
    }
}

Sample timings

Notes:

Run 1


    outside plugin - new MyContext() : 55
    outside plugin - FirstOrDefault(): 783
     inside plugin - new MyContext() : 352
     inside plugin - FirstOrDefault(): 2675

Run 2


    outside plugin - new MyContext() : 53
    outside plugin - FirstOrDefault(): 798
     inside plugin - new MyContext() : 355
     inside plugin - FirstOrDefault(): 2687

Run 3


    outside plugin - new MyContext() : 45
    outside plugin - FirstOrDefault(): 778
     inside plugin - new MyContext() : 355
     inside plugin - FirstOrDefault(): 2683

AppDomain research

After some further research in to the cost of AppDomains, there seems to be a suggestion that subsequent AppDomains have to re-JIT system DLLs and so there is an inherent start-up cost in creating an AppDomain. Is that what is happening here? I would have expected that the JIT-ing would have been on AppDomain creation, but perhaps it is EF JIT-ing when it is called?

Reference for re-JIT: http://msdn.microsoft.com/en-us/magazine/cc163655.aspx#S8

Timings sounds similar, but not sure if related: First WCF connection made in new AppDomain is very slow

Update 1

Based on @Yasser's suggestion that there is EF communication across the AppDomains, I tried to isolate this further. I don't believe this to be the case.

I have completely removed any EF reference from EF.csproj. I now have enough rep to post images, so this is the solution structure:

EF.sln

As you can see, only the plugin has a reference to Entity Framework. I have also verified that only the plugin has a bin folder with an EntityFramework.dll.

I have added a helper to verify if the EF assembly has been loaded in the AppDomain. I have also verified (not shown) that after the call to the database, additional EF assemblies (e.g. dynamic proxy) are also loaded.

So, checking if EF has loaded at various points:

  1. In Main before calling the plugin
  2. In Plugin before hitting the database
  3. In Plugin after hitting the database
  4. In Main after calling the plugin

... produces:

Main - IsEFLoaded: False
Plugin - IsEFLoaded: True
Plugin - new MyContext() : 367
Plugin - FirstOrDefault(): 2693
Plugin - IsEFLoaded: True
Main - IsEFLoaded: False

So it seems that the AppDomains are fully isolated (as expected) and the timings are the same inside the plugin.

Updated Sample code

Program.cs

class Program
{
    static void Main(string[] args)
    {
        var dir = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF.Plugin\bin\Debug");
        var evidence = new Evidence();
        var setup = new AppDomainSetup { ApplicationBase = dir };
        var domain = AppDomain.CreateDomain("other", evidence, setup);
        var pluginDll = Path.Combine(dir, "EF.Plugin.dll");
        var plugin = (IPlugin) domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");

        Console.WriteLine("Main - IsEFLoaded: " + Helper.IsEFLoaded());
        plugin.FirstPost();
        Console.WriteLine("Main - IsEFLoaded: " + Helper.IsEFLoaded());

        Console.ReadLine();
    }
}

Helper.cs

(Yeah, I wasn’t going to add another project for this…)

public static class Helper
{
    public static bool IsEFLoaded()
    {
        return AppDomain.CurrentDomain
            .GetAssemblies()
            .Any(a => a.FullName.StartsWith("EntityFramework"));
    }
}

SamplePlugin.cs

public class SamplePlugin : MarshalByRefObject, IPlugin
{
    public void FirstPost()
    {
        Console.WriteLine("Plugin - IsEFLoaded: " + Helper.IsEFLoaded());

        var watch = Stopwatch.StartNew();
        var context = new MyContext();
        watch.Stop();
        Console.WriteLine("Plugin - new MyContext() : " + watch.ElapsedMilliseconds);

        watch = Stopwatch.StartNew();
        var posts = context.Posts.FirstOrDefault();
        watch.Stop();
        Console.WriteLine("Plugin - FirstOrDefault(): " + watch.ElapsedMilliseconds);

        Console.WriteLine("Plugin - IsEFLoaded: " + Helper.IsEFLoaded());
    }
}

Update 2

@Yasser: System.Data.Entity is loaded in to the plugin only after hitting the database. Initially only the EntityFramework.dll is loaded in the plugin, but post-database other EF assemblies are loaded too:

Loaded assemblies

Zipped solution. The site only keeps files for 30 days. Feel free to suggest a better file sharing site.

Also, I am interested to know if you can verify my findings by referencing EF in the main project and seeing if the timings pattern from the original sample are reproducible.

Update 3

To be clear, it is first call timings that I am interested in analyzing which includes EF startup. On first call, going from ~800ms in a parent AppDomain to ~2700ms in a child AppDomain is very noticeable. On subsequent calls, going from ~1ms to ~3ms is hardly noticeable at all. Why is the first call (including EF startup) so much more expensive inside child AppDomains?

I’ve updated the sample to focus just on a FirstOrDefault() call to reduce the noise. Some timings for running in the parent AppDomain and running in 3 child AppDomains:

EF.vshost.exe|0|FirstOrDefault(): 768
EF.vshost.exe|1|FirstOrDefault(): 1
EF.vshost.exe|2|FirstOrDefault(): 1

AppDomain0|0|FirstOrDefault(): 2623
AppDomain0|1|FirstOrDefault(): 2
AppDomain0|2|FirstOrDefault(): 1

AppDomain1|0|FirstOrDefault(): 2669
AppDomain1|1|FirstOrDefault(): 2
AppDomain1|2|FirstOrDefault(): 1

AppDomain2|0|FirstOrDefault(): 2760
AppDomain2|1|FirstOrDefault(): 3
AppDomain2|2|FirstOrDefault(): 1

Updated Sample Code

    static void Main(string[] args)
    {
        var mainPlugin = new SamplePlugin();

        for (var i = 0; i < 3; i++)
            mainPlugin.Do(i);

        Console.WriteLine();

        for (var i = 0; i < 3; i++)
        {
            var plugin = CreatePluginForAppDomain("AppDomain" + i);

            for (var j = 0; j < 3; j++)
                plugin.Do(j);

            Console.WriteLine();
        }

        Console.ReadLine();
    }

    private static IPlugin CreatePluginForAppDomain(string appDomainName)
    {
        var dir = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF.Plugin\bin\Debug");
        var evidence = new Evidence();
        var setup = new AppDomainSetup { ApplicationBase = dir };
        var domain = AppDomain.CreateDomain(appDomainName, evidence, setup);
        var pluginDll = Path.Combine(dir, "EF.Plugin.dll");
        return (IPlugin) domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");
    }

public class SamplePlugin : MarshalByRefObject, IPlugin
{
    public void Do(int i)
    {
        var context = new MyContext();

        var watch = Stopwatch.StartNew();
        var posts = context.Posts.FirstOrDefault();
        watch.Stop();
        Console.WriteLine(AppDomain.CurrentDomain.FriendlyName + "|" + i + "|FirstOrDefault(): " + watch.ElapsedMilliseconds);
    }
}

Zipped solution. The site only keeps files for 30 days. Feel free to suggest a better file sharing site.

Upvotes: 27

Views: 2155

Answers (3)

Stajs
Stajs

Reputation: 1370

This seems to be just the cost of child AppDomains. A rather ancient post (which may no longer be relevant) suggests that there could be other considerations outside of just having to JIT-compile each child AppDomain, e.g. evaluating security policies.

Entity Framework does have a relatively high startup cost so the effects are magnified, but for comparision calling other parts of System.Data (e.g. a straight SqlDataReader) is just as horrible:

EF.vshost.exe|0|SqlDataReader: 67
EF.vshost.exe|1|SqlDataReader: 0
EF.vshost.exe|2|SqlDataReader: 0

AppDomain0|0|SqlDataReader: 313
AppDomain0|1|SqlDataReader: 2
AppDomain0|2|SqlDataReader: 0

AppDomain1|0|SqlDataReader: 290
AppDomain1|1|SqlDataReader: 3
AppDomain1|2|SqlDataReader: 0

AppDomain2|0|SqlDataReader: 316
AppDomain2|1|SqlDataReader: 2
AppDomain2|2|SqlDataReader: 0

public class SamplePlugin : MarshalByRefObject, IPlugin
{
    public void Do(int i)
    {
        var watch = Stopwatch.StartNew();
        using (var connection = new SqlConnection("Data Source=.\\sqlexpress;Initial Catalog=EF.Plugin.MyContext;Integrated Security=true"))
        {
            var command = new SqlCommand("SELECT * from Posts;", connection);
            connection.Open();
            var reader = command.ExecuteReader();
            reader.Close();
        }
        watch.Stop();

        Console.WriteLine(AppDomain.CurrentDomain.FriendlyName + "|" + i + "|SqlDataReader: " + watch.ElapsedMilliseconds);
    }
}

Even newing up a humble DataTable is inflated:

EF.vshost.exe|0|DataTable: 0
EF.vshost.exe|1|DataTable: 0
EF.vshost.exe|2|DataTable: 0

AppDomain0|0|DataTable: 12
AppDomain0|1|DataTable: 0
AppDomain0|2|DataTable: 0

AppDomain1|0|DataTable: 11
AppDomain1|1|DataTable: 0
AppDomain1|2|DataTable: 0

AppDomain2|0|DataTable: 10
AppDomain2|1|DataTable: 0
AppDomain2|2|DataTable: 0

public class SamplePlugin : MarshalByRefObject, IPlugin
{
    public void Do(int i)
    {
        var watch = Stopwatch.StartNew();
        var table = new DataTable("");
        watch.Stop();

        Console.WriteLine(AppDomain.CurrentDomain.FriendlyName + "|" + i + "|DataTable: " + watch.ElapsedMilliseconds);
    }
}

Upvotes: 4

Yaser Moradi
Yaser Moradi

Reputation: 3317

Maybe I'm wrong but with following code :

public class SamplePlugin : MarshalByRefObject, IPlugin
{
    public void Do()
    {
        using (AppDb db = new AppDb())
        {
            db.Posts.FirstOrDefault();
        }
    }
}

and these codes :

[LoaderOptimization(LoaderOptimization.MultiDomain)]
    static void Main(String[] args)
    {
        AppDomain.CurrentDomain.AssemblyLoad += CurrentDomain_AssemblyLoad;

        var dir = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF\bin\Debug");

        var evidence = new Evidence();

        var setup = new AppDomainSetup { ApplicationBase = dir };

        var domain = AppDomain.CreateDomain("Plugin", evidence, setup);

        domain.AssemblyLoad += domain_AssemblyLoad;

        var pluginDll = Path.Combine(dir, "EF.Plugin.dll");

        var anotherDomainPlugin = (IPlugin)domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");

        var mainDomainPlugin = new SamplePlugin();

        mainDomainPlugin.Do();    // To prevent side effects of entity framework startup from our test

        anotherDomainPlugin.Do(); // To prevent side effects of entity framework startup from our test

        Stopwatch watch = Stopwatch.StartNew();

        mainDomainPlugin.Do();

        watch.Stop();

        Console.WriteLine("Main Application Domain -------------------------- " + watch.ElapsedMilliseconds.ToString());

        watch.Restart();

        anotherDomainPlugin.Do();

        watch.Stop();

        Console.WriteLine("Another Application Domain -------------------------- " + watch.ElapsedMilliseconds.ToString());

        Console.ReadLine();
    }

    static void CurrentDomain_AssemblyLoad(Object sender, AssemblyLoadEventArgs args)
    {
        Console.ForegroundColor = ConsoleColor.Red;
        Console.WriteLine("Main Domain : " + args.LoadedAssembly.FullName);
    }

    static void domain_AssemblyLoad(Object sender, AssemblyLoadEventArgs args)
    {
        Console.ForegroundColor = ConsoleColor.Green;
        Console.WriteLine("Another Domain : " + args.LoadedAssembly.FullName);
    }

There is no real performance difference between main application domain and another application domain in this scenario, You get different results because your tests are wrong (-: ( At least I think they are wrong ), I've also tested main application domain by directly invoking DbContext and first or default , My times are the same and difference is between 1 - 2 milliseconds , I can't understand why my results are different than your results

Upvotes: 0

Yaser Moradi
Yaser Moradi

Reputation: 3317

You should run that test several times when you start your application

After first time , the performance difference is all about serialization of objects between your main application domain and the plugin application domain.

Notice that each communication between application domains need serialization & deserialization which costs too much.

You can see this problem while developing applications on [SQL Server / .NET CLR] stored procedures which run in separated application domain rather than sql server engine.

Upvotes: 3

Related Questions