CODEWITHSUNDEEP
c#filesystemsfilestreamstreamreadermemorystream
PArt
PArt

Reputation: 23

Difference between reading from memory and reading from a file on desk in performance

I am working on a c# project where I have some text files (they are about 30mb) The program extract those files from a game client. then stores the content in memory (Lists) to work with. so the process I am working with is:

  1. Parse game client.
  2. Save content as txt files in a directory.
  3. Use stream reader to read content line by line and store each line in lists.

But I end up with my program using about 100MB ram which is kinda a lot, cuz the program is meant to be a bot that runs multiple accounts for that game (20 accounts expected so that's about 2GB ram I believe).

Would it be better if I parse game client then save txt files and then read from the txt files when needed without storing content in memory? or would it be worse in performance?

Upvotes: 1

Views: 3179

Answers (2)

markaaronky
markaaronky

Reputation: 1305

You might look at using memory mapped files. They do what they sound like. You map a section of memory to a disk file; you access memory but you're actually touching the underlying disk file. The benefits are that you bypass some of the normal (performance-sucking) file I/O overhead you'd encounter if you used the .Net file operations, and your code looks clean because you're just doing what you'd always do--writing code against object in memory, not mucking about writing classes to read and write files.

One way you might think of memory mapped files is if you're familiar with how operating systems page in virtual memory. I'm generalizing here, but it's not a bad analogy.

Take a look at Working with memory mapped files in .Net

Upvotes: 1

Phil S
Phil S

Reputation: 846

The answer is...it depends on your use case.

If you're often referring to values that are stored within those files while the program is running, and those files are not too big, it generally makes sense to store them in memory. You get MUCH faster access than reading from disk.

One commonly used solution to this kind of issue is to use a cache (e.g. if you had more values than you can feasibly store in memory).

You can limit the size of your cache and store the most commonly used variables there. Have your program try to read from the cache first - if the value it wants is there (a hit) then job done. If the value isn't there (a miss), it will be fetched from disk and placed into the cache for future reference.

Old or less commonly used values can be evicted from the cache if it starts getting too full.

If you're running multiple instances of the same program, with common data, you you use programs like Redis as a cache that they all refer to. (i.e. one instance of 100mb, instead of 100mb x number of bots)

Upvotes: 0

Related Questions