CODEWITHSUNDEEP
c#.netmemory
Zippo
Zippo

Reputation: 16420

Storing data in memory, which is a good way?

I have a items.xml file, which is loaded into memory once the application starts. Each element in items.xml is "converted" to type Item (which has just few int & string properties), then added into a list of items. The item list should hold thousands instances of Item.

Storing data in objects (Item class in my example) is ok? Is there another way to store such data in memory?

Upvotes: 4

Views: 9704

Answers (5)

mattmc3
mattmc3

Reputation: 18325

Is your items list read-only, or do you need to handle modifications? If it's read-only, then everyone's advice thus far is fine. If your in-memory objects are modifiable, then there's a whole lot of questions left unanswered.

If you're going to handle modifications, how will you:

  1. ...handle concurrent access?
  2. ...sync your items list back to the XML document?
  3. ...expire your in-memory cache if someone modifies the underlying XML document?
  4. ...ensure that you don't lose data if the application crashes/restarts?

Using in-memory objects for rapid access is good practice, and I do it myself all the time. But it's not an easy solution if your data is volatile. In that scenario, you should use an ACID compliant database.

Upvotes: 0

Marc Gravell
Marc Gravell

Reputation: 1062640

That is fine, and as long as you ensure you don't eat all the memory should be fine. I've done this for complex systems with larger numbers.

If your data is regularly accessed by something other than the index in the list, you might want to use a dictionary to index the item, for example:

Dictionary<string, Item> lookup =
    list.ToDictionary(x => x.Code);

Then you can access with:

Item item = lookup["abc012"];

Also; if many of the string values ate repeated you can save some apace by writing your own interner;

Dictionary<string,string> interner =
    new Dictionary<string,string>();
foreach(Item item in list) {
    string s, name = item.Name;
    if(interner.TryGetValue(name, out s))
        item.Name = s;
    else
        interner.Add(name, name);
}

Which will reduce memory use by only keeping and re-using unique strings

Upvotes: 5

ChrisF
ChrisF

Reputation: 137128

Without knowing exactly what these Item objects are used for I can't say for sure, but what you describe sounds perfectly OK.

If you have 100,000's of these objects (rather than 1,000's) then a List wouldn't be the best data structure to use. There are two approaches to this. Either leave it as a list and then deal with it when it does become a problem, or fix it now.

The former means you don't have work to do now, but you have an unknown amount of work to do in the future - there will be more potential implications to a change in data structure later (more code to change, possible bugs being introduced).

The latter means a known amount of work now, but (hopefully) no work in the future as you've chosen a data structure that scales. However, beware of premature optimisation. If your list never grows beyond it's current size you've done this work for no gain.

Upvotes: 0

Carlo V. Dango
Carlo V. Dango

Reputation: 13832

First worry about getting the functionality right. Second, worry about performance. Profile the application and see where it hurts performance-wise. It may never be in your config. If, however, you suffer performance problems in this area, try to invent keys for either single or subsets of Items. This way you can populate one or several dictionaries with keys and Items (or lists of items) thereby reducing the access time to the Item in question.

Remember that these look-up dictionaries are "cheap" in that they only store references to your items rather than copies of the objects.

Upvotes: 7

Oded
Oded

Reputation: 498972

This is fine, you would be using one type of object or another anyways, so custom ones are a good option.

There are other ways to load and query XML data - ffor example using XDocument and querying it with Linq to XML.

Upvotes: 1

Related Questions