CODEWITHSUNDEEP
python-2.7google-app-enginegoogle-cloud-datastoreapp-engine-ndb
Ruslan
Ruslan

Reputation: 70

ndb put() work but not update result at the same time just after 15-30 seconds

I have very interesing issue(for me exactly), I have query and in this query i have list row:

class TestList(ndb.Model):
    testing_list = ndb.StringProperty(repeated=True)
    list_id = ndb.IntegerProperty()

Also I have API methods for changing testing_list by PATCH request. Code for this:

@app.route('/list/change', methods=['PATCH'])
def list_change():
    list_id = request.form['id']
    list_elements = request.form['elements']
    query = TestList.query(TestList.list_id == int(list_id))
    try:
        query.fetch()[0].list_id
    except IndexError:
        return 'Error', 400
    new_elements = str(list_elements).replace(' ', '').split(',')
    query.fetch()[0].testing_list = [element for element in new_elements if element in query.fetch()[0].testing_list]
    query.fetch()[0].put()

    testing_list_extend(query.get(), new_elements)
    return 'Success', 200

@ndb.transactional
def testing_list_extend(list_key, new_elements):
    for element in new_elements:
        query = TestList.query(ancestor=list_key.key)
        if element not in query.fetch()[0].testing_list:
            query.fetch()[0].testing_list.append(element)
            query.fetch()[0].put()
    return '200'

On input I get string like 'Element1, Element2' this is elements in body request and id like '1'. So after I parse string and make list. After I want add new unique elements in testing_list. On this part I have bug: sometimes, when I add new elements and get testing_list by GET request I get empty list, but for a 15-30 seconds I get list which I wanted see some time ago. For example in body PATCH request:

id = '1'
elements = 'Element1, Element2'

What response I waiting by getting testing_list:

[Element1, Element2]

What I get very often:

[Element1, Element2]

What I get very rare(bug as I think):

[]

Upvotes: 0

Views: 211

Answers (2)

Temu
Temu

Reputation: 879

The problem was in ndb caching.

Writing functional test for this issue and find out that row on Datastore in google.cloud was updated but in the same time a GET request was did and got old date, so in the GET method was put ndb.get_context().clear_cache() and this worked fine.

Upvotes: 1

Alex
Alex

Reputation: 5276

There are a few things wrong here, but i think the cause of your issue is the multiple puts and fetches you do. This will be much better if you can use the TestLists key instead of TestList.list_id. That way your function would look something like this:

@app.route('/list/change', methods=['PATCH'])
def list_change():
    list_id = request.form['id']
    list_elements = request.form['elements']
    new_elements = str(list_elements).replace(' ', '').split(',')
    try:
        testing_list_extend(ndb.Key(TestList, long(list_id)), new_elements)
        return 'Success', 200
    except Exception as e:
        return e.message, 400

@ndb.transactional
def testing_list_extend(list_key, new_elements):
    test_list = list_key.get()
    if test_list is None:
        raise Exception('Test List ID does not exist')
    l = []
    l.extend(entity.testing_list)  # the existing list
    l.extend(new_elements)  # the append the new_elements
    entity.testing_list = list(set(l))  # remove duplicates
    entity.put()

Otherwise, try doing it like this:

@app.route('/list/change', methods=['PATCH'])
def list_change():
    list_id = request.form['id']
    list_elements = request.form['elements']
    new_elements = str(list_elements).replace(' ', '').split(',')
    try:
        # Only return the Key, to be used in the transaction below
        query = TestList.query(TestList.list_id == int(list_id)).fetch(2, keys_only=True)
        if len(query) == 0:
            raise Exception("Found no 'TestList' with list_id == %s" % list_id)
        # Double check for duplicates
        elif len(query) == 2:
            raise Exception("Found more than one 'TestList' with list_id == %s" % list_id)
        testing_list_extend(query[0], new_elements)
        return 'Success', 200
    except Exception as e:
        return e.message, 400

@ndb.transactional
def testing_list_extend(list_key, new_elements):  # same
    test_list = list_key.get()
    if test_list is None:
        raise Exception('Test List ID does not exist')
    l = []
    l.extend(entity.testing_list)  # the existing list
    l.extend(new_elements)  # the append the new_elements
    entity.testing_list = list(set(l))  # remove duplicates
    entity.put()

Upvotes: 1

Related Questions