CODEWITHSUNDEEP
rubymemory-managementgarbage-collection
unexist
unexist

Reputation: 2528

What are your strategies to keep the memory usage low?

Ruby is truly memory-hungry - but also worth every single bit.

What do you do to keep the memory usage low? Do you avoid big strings and use smaller arrays/hashes instead or is it no problem to concern about for you and let the garbage collector do the job?

Edit: I found a nice article about this topic here - old but still interesting.

Upvotes: 30

Views: 5559

Answers (17)

severin
severin

Reputation: 10258

  1. When deploying a Rails/Rack webapp, use REE or some other copy-on-write friendly interpreter.
  2. Tweak the garbage collector (see https://www.engineyard.com/blog/tuning-the-garbage-collector-with-ruby-1-9-2 for example)
  3. Try to cut down the number of external libraries/gems you use since additional code uses memory.
  4. If you have a part of your app that is really memory-intensive then it's maybe worth rewriting it in a C extension or completing it by invoking other/faster/better optimized programs (if you have to process vast amounts of text data, maybe you can replace that code with calls to grep, awk, sed etc.)

Upvotes: 5

tokhi
tokhi

Reputation: 21618

Ruby developers are quite lucky since they don’t have to manage the memory themselves.

Be aware that ruby allocates objects, for instance something as simple as 

100.times{ 'foo' }

allocates 100 string objects (strings are mutable and each version requires its own memory allocation).

Make sure that if you are using a library allocating a lot of objects, that other alternatives are not available and your choice is worth paying the garbage collector cost. (you might not have a lot of requests/s or might not care for a few dozen ms per requests).

Creating a hash object really allocates more than an object, for instance

{'joe' => 'male', 'jane' => 'female'}

doesn’t allocate 1 object but 7. (one hash, 4 strings + 2 key strings)

If you can use symbol keys as they won’t be garbage collected. However because they won’t be garbage collected you want to make sure to not use totally dynamic keys like converting the username to a symbol, otherwise you will ‘leak’ memory.

Example: Somewhere in your app, you apply a to_sym on an user’s name like :

hash[current_user.name.to_sym] = something

When you have hundreds of users, that’s could be ok, but what is happening if you have one million of users ? Here are the numbers :

ruby-1.9.2-head >
# Current memory usage : 6608K
# Now, add one million randomly generated short symbols
ruby-1.9.2-head > 1000000.times { (Time.now.to_f.to_s).to_sym }

# Current memory usage : 153M, even after a Garbage collector run.

# Now, imagine if symbols are just 20x longer than that ?
ruby-1.9.2-head > 1000000.times { (Time.now.to_f.to_s * 20).to_sym }
# Current memory usage : 501M

Be aware to never convert non controlled arguments in symbol or check arguments before, this can easily lead to a denial of service.

Also remember to avoid nested loops more than three levels deep because it makes the maintenance difficult. Limiting nesting of loops and functions to three levels or less is a good rule of thumb to keep the code performant.

Here are some links in regards:

http://merbist.com

http://blog.monitis.com

Upvotes: 6

Kazuki Ohta
Kazuki Ohta

Reputation: 1441

Replacing malloc(3) implementation to jemalloc will immediately decrease your memory consumption up to 30%. I've created 'jemalloc' gem to achieve this instantly.

Upvotes: 1

Martin Velez
Martin Velez

Reputation: 1409

When possible, use arrays instead of other data structures. Try not to use floats when integers will do.

Be careful when using gem/library methods. They may not be memory optimized. For example, the Ruby PG::Result class has a method 'values' which is not optimized. It will use a lot of extra memory. I have yet to report this.

Upvotes: 1

Oscar Del Ben
Oscar Del Ben

Reputation: 4515

Something to keep in mind is the life cycle of your objects. If you're objects are not passed around that much, the garbage collector will eventually kick in and free them up. However, if you keep referencing them it may require some cycles for the garbage collector to free them up. This is particularly true in Ruby 1.8, where the garbage collector uses a poor implementation of the mark and sweep technique.

You may run into this situation when you try to apply some "design patterns" like decorator that keep objects in memory for a long time. It may not be obvious when trying example in isolation, but in real world applications where thousands of objects are created at the same time the cost of memory growth will be significant.

Upvotes: 1

Orlando
Orlando

Reputation: 9692

dont use a lot of symbols, they stay in memory until the process gets killed.. this because symbols never get garbage collected.

Upvotes: 0

Grant Hutchins
Grant Hutchins

Reputation: 4409

Avoid code like this:

str = ''
veryLargeArray.each do |foo|
  str += foo
  # but str << foo is fine (read update below)
end

which will create each intermediate string value as a String object and then remove its only reference on the next iteration. This junks up the memory with tons of increasingly long strings that have to be garbage collected.

Instead, use Array#join:

str = veryLargeArray.join('')

This is implemented in C very efficiently and doesn't incur the String creation overhead.

UPDATE: Jonas is right in the comment below. My warning holds for += but not <<.

Upvotes: 2

unbeknown
unbeknown

Reputation:

I'm using Python, but I guess the strategies are similar.

I try to use small functions/methods, so that local variables get automatically garbage collected when you return to the caller.

In larger functions/methods I explicitly delete large temporary objects (like lists) when they are no longer needed. Closing resources as early as possible might help too.

Upvotes: 1

Orion Edwards
Orion Edwards

Reputation: 123622

The only thing we've ever had which has actually been worth worrying about is RMagick.

The solution is to make sure you're using RMagick version 2, and call Image#destroy! when you're done using your image

Upvotes: 2

Kevin Haines
Kevin Haines

Reputation: 2552

Take a look at Small Memory Software - Patterns for Systems with Limited Memory. You don't specify what sort of memory constraint, but I assume RAM. While not Ruby-specific, I think you'll find some useful ideas in this book - the patterns cover RAM, ROM and secondary storage, and are divided into major techniques of small data structures, memory allocation, compression, secondary storage, and small architecture.

Upvotes: 2

Mark Kegel
Mark Kegel

Reputation: 4510

  1. Choose date structures that are efficient representations, scale well, and do what you need.
  2. Use algorithms that work using efficient data structures rather than bloated, but easier ones.
  3. Look else where. Ruby has a C bridge and its much easier to be memory conscious in C than in Ruby.

Upvotes: 6

s3v1
s3v1

Reputation: 2933

I try to keep arrays & lists & datasets as small as possible. The individual object do not matter much, as creation and garbage collection is pretty fast in most modern languages.

In the cases you have to read some sort of huge dataset from the database, make sure to read in a forward/only manner and process it in little bits instead og loading everything into memory first.

Upvotes: 0

Toby Hede
Toby Hede

Reputation: 37133

I really don't think it matters all that much. Making your code less readable in order to improve memory consumption is something you should only ever do if you need it. And by need, I mean have a specific case for the performance profile and specific metrics that indicate that any change will address the issue.

If you have an application where memory is going to be the limiting factor, then Ruby may not be the best choice. That said, I have found that my Rails apps generally consume about 40-60mb of RAM per Mongrel instance. In the scheme of things, this isn't very much.

You might be able to run your application on the JVM with JRuby - the Ruby VM is currently not as advanced as the JVM for memory management and garbage collection. The 1.9 release is adding many improvements and there are alternative VM's under development as well.

Upvotes: 8

Loren Pechtel
Loren Pechtel

Reputation: 9083

Other than in extreme cases memory usage isn't something to worry about. The time you spend trying to reduce memory usage will buy a LOT of gigabytes.

Upvotes: 2

Ben Scofield
Ben Scofield

Reputation: 6418

I've found Phusion's Ruby Enterprise Edition (a fork of mainline Ruby with much-improved garbage collection) to make a dramatic difference in memory usage... Plus, they've made it extraordinarily easy to install (and to remove, if you find the need).

You can find out more and download it on their website.

Upvotes: 9

John Zwinck
John Zwinck

Reputation: 249103

I'm pretty new at Ruby, but so far I haven't found it necessary to do anything special in this regard (that is, beyond what I just tend to do as a programmer generally). Maybe this is because memory is cheaper than the time it would take to seriously optimize for it (my Ruby code runs on machines with 4-12 GB of RAM). It might also be because the jobs I'm using it for are not long-running (i.e. it's going to depend on your application).

Upvotes: 1

alexmac
alexmac

Reputation: 4201

I am not a ruby developer but I think some techniques and methods are true of any language:

Use the minimum size variable suitable for the job
Destroy and close variables and connections when not in use
However if you have an object you will need to use many times consider keeping it in scope Any loops with manipulations of a big string dp the work on a smaller string and then append to bigger string

Use decent (try catch finally) error handling to make sure objects and connections are closed

When dealing with data sets only return the minimum necessary

Upvotes: 3

Related Questions