CODEWITHSUNDEEP
c#performanceforeach
Srini
Srini

Reputation: 126

Performance issue with search of large in memory collection

I wrote a query to find node in nodedata from transitiondata but it is taking quite a long time to come out of that loop since it has 4 million records.

What we have : 1. Transition data(Collection) which will have from and to node.
2. Node data(Collection) which will have key which is equals to form or to node from Transition data(Collection)

What is required out of these collections: 1. Collection which should have Transition Data(from, to) and the corresponding nodes from Node data(from key) and (to key)

The code what i wrote works fine, but it takes lot of time to execute. Below is the code.

foreach (var trans in transitions)
        {
            string transFrom = trans.From;
            string transTo = trans.To;

            var fromNodeData = nodeEntitydata.Where(x => x.Key == transFrom).FirstOrDefault();
            var toNodeData = nodeEntitydata.Where(x => x.Key == transTo).FirstOrDefault();

            if (fromNodeData != null && toNodeData != null)
            {
                //string fromSwimlane = fromNodeData.Group;
                //string toSwimlane = toNodeData.Group;
                string dicKey = fromNodeData.sokey + toNodeData.sokey;
                if (!dicTrans.ContainsKey(dicKey))
                {
                    soTransition.Add(new TransitionDataJsonObject
                    {
                        From = fromNodeData.sokey,
                        To = toNodeData.sokey,
                        FromPort = fromPortIds[0],
                        ToPort = toPortIds[0],
                        Description = "SOTransition",
                        IsManual = true
                    });
                    dicTrans.Add(dicKey, soTransition);
                }
            }
        }

That is the loop which takes time to execute. I know the problem is in that two Where clause. Because transitions will have 400k and nodeEntitydata will have 400k. Can someone help me on this?

Upvotes: 1

Views: 83

Answers (2)

Doctor Jones
Doctor Jones

Reputation: 21664

It looks like nodeEntitydata is just a normal collection. The problem you're facing is that performing a Where on an in memory collection has linear performance, and you've got a lot of records to process.

What you need is a Dictionary. This is much more efficient for searching large collections, because it uses a binary tree to do the search rather than a linear search.

If nodeEntitydata isn't already a Dictionary, you can create a Dictionary from it like this:

var nodeEntitydictionary = nodeEntitydata.ToDictionary(n => n.Key);

You can then consume the dictionary like this:

var fromNodeData = nodeEntitydictionary[transFrom];
var toNodeData = nodeEntitydictionary[transTo];

Creating the Dictionary will be fairly slow, so make sure you only do it once at the point where you populate nodeEntitydata. If you have to keep re-instantiating the Dictionary too frequently then you won't see much of a performance benefit, so make sure you reuse it as much as possible.

Upvotes: 0

DrKoch
DrKoch

Reputation: 9782

Use direct access to the dictionary entry:

var fromNodeData = nodeEntitydata[transFrom];
var toNodeData = nodeEntitydata[transTo];

Upvotes: 1

Related Questions