CODEWITHSUNDEEP
c#ravendbdocument-database
NPehrsson
NPehrsson

Reputation: 1548

RavenDb importing data from different sources like xml, cvs

We have a system which will import alot of data. The data can come with different types of xml files and are right now mapped to c# objects. Now we want by the name (which will be unique, but will not be the Id) update existing objects that are located by name, or create new documents if no matching name.

How can we accomplish this in the most efficient way?

Upvotes: 2

Views: 539

Answers (1)

Pure.Krome
Pure.Krome

Reputation: 86957

If you didn't care about checking for existing, then you could do a Bulk Insert.

Besides that, you will need to load each one by Name (because I'm assuming that is unique, as you said).

Like this..

using (var documentSession = documentStore.OpenSession())
{
    // We are doing an import, so allow us to do LOTS of queries to the db.
    documentSession.Advanced.MaxNumberOfRequestsPerSession = int.MaxValue;

    foreach (var foo in newFoosToImport())
    {
        var existingFoo = documentSession
            .Query<Foo>()
            .SingleOrDefault(x => x.Name == foo.Name);

        if (existingFoo == null)
        {
            // Doesn't exist, so just save this new data.
            documentSession.Store(foo);
        }
        else
            // Map the required new data to the existing object.       
            MapNewDataToExistingObject(foo, existingFoo);

            // Now save the existing.
            documentSessionStore(existingFoo);           
        }
    }

    // Commit these changes.
    documentSession.SaveChanges();
}

Another alternative to this code approach would be to reduce the number of roundtrips to the db by batching up the Names. Meaning, you pass a list of names to the Where clause. So maybe pass in 10 or 30 Names. If it exists, then update those records, else insert.

Now if you have a large data set, you might consider batching your SaveChanges.

eg.

if (storeCount % 1024 == 0)
{
    documentSession.SaveChanges();
}

So every 1024 Stores, you then SaveChanges. Not sure if that helps or not but it's an idea. (Tip: If you do do that, then make sure you have one more SaveChanges after the loop so you commit the last set of data in the loop).

Upvotes: 2

Related Questions