CODEWITHSUNDEEP
ravendb
Stephen Patten
Stephen Patten

Reputation: 6363

How to add a couple of million documents to RavenDB - Embedded

I have installed the latest embedded binaries from NuGet and am using this code to store a 'product' poco. After a bit, the process dies with an OutOfMemoryException. Is storing this much data out of scope for Raven?

Thank you. Stephen

var store = new EmbeddableDocumentStore { DataDirectory = @"C:\temp\ravendata", UseEmbeddedHttpServer = true };
store.Initialize();

using (var session = store.OpenSession())
{
    foreach (var item in Parsers.GetProducts().ToList())
    {
        session.Store(item);

    }
    session.SaveChanges();
    //var rdbList = session.Query<Product>().ToList();
}


[Serializable]
public class Product
{
    public decimal ProductId { get; set; }

    public string ItemNum { get; set; }

    public string ProductName { get; set; }

    public string BrandName { get; set; }

    public string UOM { get; set; }

    public string AveWeight { get; set; }

    public string CasePack { get; set; }

    public string PackageRemarks { get; set; }

    public decimal Price { get; set; }

    public string SupplierName { get; set; }

    public string Url { get; set; }

    public bool IsSpecialOrderItem { get; set; }

    public bool IsSpecialPriceItem { get; set; }

    public bool IsRebateItem { get; set; }

    public bool IsTieredPricingItem { get; set; }

    public bool IsOfflineSupplierItem { get; set; }

    public string Catalog { get; set; }

    public decimal CatalogId { get; set; }

    public decimal CategoryId { get; set; }

    public decimal PriceGroupId { get; set; }

    public decimal OffineSupplierId { get; set; }

    public string ManufactureName { get; set; }

    public string ManufactureNum { get; set; }

    public string Upc { get; set; }

    public string Info { get; set; }

    public string INFO2 { get; set; }
}

Upvotes: 1

Views: 451

Answers (2)

Daniel Lang
Daniel Lang

Reputation: 6839

No, RavenDB is perfectly fine with this amount of data. Unless you don't RunInMemory an EmbeddedDocumentStore is pretty much the same as a standalone server, just without the http overhead and direct access to the database from the client.

Given your code, you want to make sure that you store your documents in batches, e.g. 1024 for each session. Another thing you want to make sure is, that your GetProducts() method returns an IEnumerable and yields the items in a proper ETL way of doing things.

Upvotes: 2

Adam
Adam

Reputation: 3665

How big of a batch are you doing? It looks like people have success with 256 batch sizes. Seems like much more causes timeouts and memory exceptions.

*EDIT: It sounds like it is also recommended to make a new session per batch so as to not keep the session open too long which can cause timeout errors.

Upvotes: 2

Related Questions