CODEWITHSUNDEEP
pythonmemoryheap-memory
lb lb
lb lb

Reputation: 91

How to dynamically allocate memory in Python

Is there any method in python, that I can use to get a block of memory from the heap ,and use a variable to reference it. Just like the keyword "new" , or the function malloc() in other languages:

Object *obj = (Object *) malloc(sizeof(Object)); 
Object *obj = new Object();

In the project, my program is waiting to receive some data in uncertain intervals and with a certain length of bytes when correct.

I used to it like this:

void receive()// callback
{
  if(getSize()<=sizeof(DataStruct))
  {
   DataStruct *pData=malloc(sizeof(DataStruct));
   if(recvData(pData)>0)
   list_add(globalList,pData);
  }   
}

void worker()
{
  init()
  while(!isFinish)
 {
  dataProcess(globalList);
 }

}

Now, I want to migrate these old project to python, and I tried to do it like this:

def reveive():
 data=dataRecv()
 globalList.append(data)

However, I get the all item in the list are same, and equal to the latest received item. It is obvious that all the list items are point to the same memory adress, and I want to get a new memory adress each the function is called.

Upvotes: 9

Views: 49532

Answers (3)

Viktor Engelmann
Viktor Engelmann

Reputation: 152

Keep in mind that interpreted languages usually don't flatten the types as compiled languages do. The memory layout is (probably) completely different than in the raw data. Therefore, you cannot simply cast raw data to a class instance or vice versa. You have to read the raw data, interpret it and fill your objects manually.

Upvotes: 1

clocker
clocker

Reputation: 1366

The equivalent of "new" in python is to just use a constructor eg:

new_list = list() # or [] - expandable heterogeneous list
new_dict = dict() # expandable hash table
new_obj = CustomObject() # assuming CustomObject has been defined

Since you are porting from C, some things to note. Everything is an object in python including integers, and most variables are just references, but the rules for scalar variables such as integers and strings are different from containers, eg:

a = 2   # a is a reference to 2
b = a   # b is a reference to 'a'
b = 3   # b now points to 3, while 'a' continues to point to 2

However:

alist = ['eggs', 2, 'juice']  # alist is reference to a new list
blist = alist # blist is a reference; changing blist affects alist
blist.append('coffee') # alist and blist both point to 
                       # ['eggs', 2, 'juice', 'coffee']

You can pre-allocate sizes, if you'd like but it often doesn't buy you much benefit in python. The following is valid:

new_list4k = [None]*4096 # initialize to list of 4096 None's 
new_list4k = [0]*4096    # initialize to 4096 0's
big_list = []
big_list.extend(new_list4k) # resizes big_list to accomodate at least 4k items

If you want to ensure memory leaks do not occur, use local variables as often as possible, eg, within a function so as things go out of scope you don't have to worry.

For efficient vectorized operations (and much lower memory footprint) use numpy arrays. 

import numpy as np
my_array = np.zeros(8192) # create a fixed array length of 8K elements
my_array += 4             # fills everything with 4

My added two cents: I'd probably start by asking what your primary goal is. There is the pythonic ways of doing things, while trying to optimize for speed of program execution or minimum memory footprint. And then there is the effort of trying to port a program in as little time as possible. Sometimes they all intersect but more often, you will find the pythonic way to be quick to translate but with higher memory requirements. Getting higher performance out of python will probably take focused experience. Good luck!

Upvotes: 8

Basile Starynkevitch
Basile Starynkevitch

Reputation: 1

You should read the Python tutorial.

You can create lists, dictionaries, objects and closures in Python. All these live in the (Python) heap, and Python has a naive garbage collector (reference counting + marking for circularity).

(the Python GC is naive because it does not use sophisticated GC techniques; hence it is slower than e.g. Ocaml or many JVM generational copying garbage collectors; read the GC handbook for more; however the Python GC is much more friendly to external C code)

Upvotes: 3

Related Questions