CODEWITHSUNDEEP
pythonbuffer
Bain
Bain

Reputation: 844

Create a buffer in python

I do not have much experience with Python and need to figure out a few things regarding buffering:

I want to generate a list in which I can allocate a certain amount of "buffer" space (do not know the specifics on this), to generate a list of integers. If the 'buffer' gets full I'm assuming it needs a flush command? Or how would you clear the buffer to continue putting things into that buffer?

Here is a sample of my code: 

for i in range(0,500):
    randoms = random.randint(0,100000)
    looplist.append(randoms)

What I would want is in looplist, to be a sort of buffer I assume? in which if the maximum buffer space in looplist gets full, it would need to clear out (is there a pause during that time? or what happens) the list to continue re-generating integers to that list/ buffer.

Part 2 question: Would like explanation as simple of how a buffer could work for python? or does memory management of python just disable need to allocate own buffers? (can we still do it if we want too?)

I will edit my question if it seems to broad scope, trying to be as descriptive as I know how.

Upvotes: 7

Views: 24792

Answers (3)

Alok Singhal
Alok Singhal

Reputation: 96121

If I understand correctly, you are asking if Python will manage a list's memory for you. The answer is yes, you can keep appending items to the list and Python will grow the list as necessary (up to a point of course -- you will run out of memory if you keep appending indefinitely).

I think you are also interested in a way to limit this, so you could be sure that your list doesn't grow beyond a limit. If that is the case, you might want to look at collections.deque objects.

For example:

import random
from collections import deque

def test_queue(max_size=10):
    d = deque([], max_size)
    for i in xrange(1, 2*max_size):
        r = random.randint(0, 100000)
        d.append(r)
        print 'i=%d, len=%d r=%d' % (i, len(d), r)

    while d:
        print d.popleft()

The above function will make sure that the deque object you create will not exceed max_size elements. If you add more elements to the deque, the oldest elements will automatically be removed. Here is an example run of the above function:

i=1, len=1 r=83870
i=2, len=2 r=12432
i=3, len=3 r=87084
i=4, len=4 r=3485
i=5, len=5 r=12237
i=6, len=6 r=81401
i=7, len=7 r=24990
i=8, len=8 r=21391
i=9, len=9 r=34153
i=10, len=10 r=63651
i=11, len=10 r=96305
i=12, len=10 r=46671
i=13, len=10 r=19288
i=14, len=10 r=40170
i=15, len=10 r=45399
i=16, len=10 r=94032
i=17, len=10 r=57749
i=18, len=10 r=68440
i=19, len=10 r=59726
63651
96305
46671
19288
40170
45399
94032
57749
68440
59726

Upvotes: 0

Thorsten Kranz
Thorsten Kranz

Reputation: 12755

In Python, you don't have to care about memory management. You don't need to reserve "buffers", just assign the variables you want, and use them.

If you assign too many (for your RAM), you might run into MemoryErrors, indicating you don't have enough memory.

You could use a Queue as some kind of buffer, though, e.g.

import random
from Queue import Queue, Full

q = Queue(100)

for i in range(10):
    #fill queue
    while True:
        try:
            q.put_nowait(random.randint(1,10))
        except Full:
            break
    #take some values from queue
    print "Round", i,
    number_of_values_to_get = random.randint(0,20)
    print "getting %i values." % number_of_values_to_get
    for j in range(number_of_values_to_get):
        value = q.get()
        print "  got value", value

I create a Queue of size 100, i.e., a maximum of 100 entries can be stored in it. If you try to put on a full queue, it will raise the corresponding exception, so you better catch it. (This is only True if you use put_nowait or put with a timeout, just look at the docs for more details.)

In this example, for ten rounds, I fill a "buffer" (queue) of 100 random integers. Then, I pick between 0 and 20 values from this buffer and print them.

I hope this helps. The main use-case for queues is multithreaded program execution, they are thread-safe. So maybe you'll want to fill the queue from one thread and read it from another. If you don't want multithreading, collections.deque might be a faster alternative, but they are not thread-safe.

Upvotes: 4

Aditya Sihag
Aditya Sihag

Reputation: 5167

use a queue / list and make indices fall off on first-in first-out basis whenever the queue / list size is sufficiently large

Upvotes: 1

Related Questions