CODEWITHSUNDEEP
pythondictionarymemoryfreeze
loyin
loyin

Reputation: 33

python2.7.6 Initialize a big dictionary but Memory would be used up

Excuse my English,

python version:2.7.6 desktop: 4 cpu-core 8G memory script1:

a = {}
a['test1'] = 12345
a['test2'] = 12456
........
and so on
........
a['test4075096'] = 45637

script2:

for i in range(0,4075096):
    a['test' + str(i)] = i

result

When I run the script2, it complete very quickly When I run the script1, it needs large mem and cpu,and my desktop stuck

So does anyone know the reasons behind this phenomenon

Upvotes: 3

Views: 58

Answers (1)

Kasravnd
Kasravnd

Reputation: 107287

It's because of that in first code, python has to read your code line by line and load CONST values to memory, while in second part you already have specified the values which would be assigned,and python will created them in memory. So all that python needs to do is iterating over the range object and assign the values to keys.

You can see this behaviour by calling dis.dis() on your functions which demonstrates the relative bytecodes for you:

>>> def foo1():
...   a = {}
...   a['test1'] = 12345
...   a['test2'] = 12456
... 

>>> import dis
>>> 
>>> 
>>> dis.dis(foo1)
  2           0 BUILD_MAP                0
              3 STORE_FAST               0 (a)

  3           6 LOAD_CONST               1 (12345)
              9 LOAD_FAST                0 (a)
             12 LOAD_CONST               2 ('test1')
             15 STORE_SUBSCR        

  4          16 LOAD_CONST               3 (12456)
             19 LOAD_FAST                0 (a)
             22 LOAD_CONST               4 ('test2')
             25 STORE_SUBSCR        
             26 LOAD_CONST               0 (None)
             29 RETURN_VALUE        
>>> 

>>> def foo2():
...   a = {}
...   for i in range(1,10):
...        a['test + str(i)'] = i
... 
>>> dis.dis(foo2)
  2           0 BUILD_MAP                0
              3 STORE_FAST               0 (a)

  3           6 SETUP_LOOP              33 (to 42)
              9 LOAD_GLOBAL              0 (range)
             12 LOAD_CONST               1 (1)
             15 LOAD_CONST               2 (10)
             18 CALL_FUNCTION            2
             21 GET_ITER            
        >>   22 FOR_ITER                16 (to 41)
             25 STORE_FAST               1 (i)

  4          28 LOAD_FAST                1 (i)
             31 LOAD_FAST                0 (a)
             34 LOAD_CONST               3 ('test + str(i)')
             37 STORE_SUBSCR        
             38 JUMP_ABSOLUTE           22
        >>   41 POP_BLOCK           
        >>   42 LOAD_CONST               0 (None)
             45 RETURN_VALUE        
>>> 
>>>

If you increase the assignment you can see that the relative bytecode will increases as well :

>>> def foo1():
...    a = {}
...    a['test1'] = 12345
...    a['test2'] = 12456
...    a['test3'] = 12457
... 
>>> dis.dis(foo1)
  2           0 BUILD_MAP                0
              3 STORE_FAST               0 (a)

  3           6 LOAD_CONST               1 (12345)
              9 LOAD_FAST                0 (a)
             12 LOAD_CONST               2 ('test1')
             15 STORE_SUBSCR        

  4          16 LOAD_CONST               3 (12456)
             19 LOAD_FAST                0 (a)
             22 LOAD_CONST               4 ('test2')
             25 STORE_SUBSCR        

  5          26 LOAD_CONST               5 (12457)
             29 LOAD_FAST                0 (a)
             32 LOAD_CONST               6 ('test3')
             35 STORE_SUBSCR        
             36 LOAD_CONST               0 (None)
             39 RETURN_VALUE

Upvotes: 2

Related Questions