Generic Import method for EF

Question

I would like to make a generic method to import data into my application.

For example, say I have:

private static async Task<int> ImportAccount(string filename)
{
    var totalRecords = await GetLineCount(filename);
    var ctx = new AccountContext();
    var count = 0;
    var records = 0;
    using (var stream = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    {
        using (var reader = new StreamReader(stream, Encoding.UTF8))
        {
            string line;
            while ((line = await reader.ReadLineAsync()) != null)
            {
                var data = line.Split('\t');
                var acc = new Account(data);
                await ctx.Accounts.AddAsync(acc);
                // need this to avoid using all the memory
                // maybe there is a smarter or beter way to do it
                // with 10k it uses about 500mb memory, 
                // files have million rows+
                if (count % 10000 == 1)
                {
                    records += result = await ctx.SaveChangesAsync();
                    if (result > 0)
                    {
                        ctx.Dispose();
                        ctx = new AccountContext();
                    }
                }
                count++;
            }
        }
    }
    await ctx.SaveChangesAsync();
    ctx.Dispose();
    return records;
}

In the above example I am importing data from a tab delimited file into the Accounts db.

Then I have properties, lands, and a whole lot of other db's I need to import.

Instead of having to make a method for each db like the above, I would like to make something like:

internal static readonly Dictionary<string, ??> FilesToImport = new Dictionary<string, ??>
{
    { "fullpath to file", ?? would be what I need to pass to T }
    ... more files ...
};
private static async Task<int> Import<T>(string filename)

Where T would be the DB in question.

All my classes have 1 thing in common, they all have a constructor that takes a string[] data.

But I have no idea how I could make a method that I would be able to accept:

private static async Task<int> Import<T>(string filename)

And then be able to do a:

var item = new T(data);
await ctx.Set<T>().AddAsync(item);

And if I recall correctly, I would not be able to instantiate T with a parameter.

How could I make this generic Import method and is it possible to achieve?

Answer 1

C# have only new() restriction for generic type arguments. But unfortunately it's not possible to force a type to have a constructor with parameters.

One workaround for this is to define an interface like this:

interface IImportedEntity<T>
// where T: YourBaseClass
{
    T Init(string[] data);
}

In this case all implementation classes will have to implement such method:

class Account : /*YourBaseClass*/ IImportedEntity<Account>
{
    public Account()
    {
        // for EF
    }

    // can be made private or protected
    public Account(string[] data)
    {
        // your code
    }


    // public Account Init(string[] data) => { /*populate current instance*/  return this;};
    // can be implemented in base class
    public Account Init(string[] data) => new Account(data);
}

Finally you can restrict your generic Import method to deal only with imported entities:

private static async Task<int> Import<T>(string filename) 
      where T: class, IImportedEntity<T>, new()
{
    ....
    var item = new T();
    item = item.Init(data);
    await ctx.Set<T>().AddAsync(item);
    ...
}

Note, if you still want to use this with a dictionary it will be needed to use reflection(example).

Answer 2

The easiest way to accomplish this is to pass a generic function that accepts the string line or a string array of split values and returns an object with values set. Use the ctx.AddAsync() method which supports generics and add the entity to the correct set.

private static async Task<int> Import<T>(string filename, Func<string, T> transform) where T : class
{
    var totalRecords = await GetLineCount(filename);
    var ctx = new AccountContext();
    var count = 0;
    var records = 0;
    using (var stream = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    {
        using (var reader = new StreamReader(stream, Encoding.UTF8))
        {
            string line;
            while ((line = await reader.ReadLineAsync()) != null)
            {
                var data = line.Split("\t");
                var entity = transform(data);
                await ctx.AddAsync(entity);
                if (count % 10000 == 1)
                {
                    records += result = await ctx.SaveChangesAsync();
                    if (result > 0)
                    {
                        ctx.Dispose();
                        ctx = new AccountContext();
                    }
                }
                count++;
            }
        }
    }
    await ctx.SaveChangesAsync();
    ctx.Dispose();
    return records;
}

// Usage

Import(filename, splits => {
   / * do whatever you need to transform the data */
   return new Whatever(splits);
})

Since generic types cannot be constructed by passing a parameter, you will have to use a function as the second type in the dictionary.

Dictionary<string, Func<string, object>> FilesToImport = new Dictionary<string, Func<string, object>>{
  { "fullpath to file", data => new Account(data) },
  { "fullpath to file", data => new Whatever(data) },
  { "fullpath to file", data => new Whatever2(data) },
}