CODEWITHSUNDEEP
pythondjangodjango-modelsdjango-queryset
derrend
derrend

Reputation: 4636

Why am I seeing duplicate model objects in Django when filtering? python

file = open('debug.txt', 'w')
db_objects = model.objects.all()

    for i in db_objects:
        db_filtered = model.objects.filter(input_address_db=i.input_address_db)
        f.write('filtered_results = %s' % db_filtered)
        ...perform operations...

here is the output of the file after the first cycle:

db_filtered = [<model: model object>, <model: model object>, <model: model object>, <model: model object>]

I checked ahead though and know that there should be only three but as you can see the debug text shows four.

Two of the above objects are the same one duplicated. I know this because I get an integrity error when I try to transfer the four objects into a new database.

The created_on dates are the primary key so it's also surprising to see the same result twice because I thought an error would be thrown, I now realise that perhaps you might want the same objects repeated in a queryset which is why there is no error but sill I'm having a bit of trouble getting my head around what is actually going on here.

Thanks in advance for any help, it's much appreciated :)

Upvotes: 1

Views: 111

Answers (2)

derrend
derrend

Reputation: 4636

I appreciate the help with this everybody and your answers helped me a lot, I just want to clarify what the problem ultimately turned out to be -

class bitcoin_deposit(models.Model):
    input_address_db = models.CharField(max_length=34, null=False, blank=False)
    deposit_amount_db = models.DecimalField(max_digits=16, decimal_places=8, validators=[MaxValueValidator(21000000)])
    return_address_db = models.CharField(max_length=34, null=False, blank=False)
    confirmations_db = models.IntegerField(validators=[MinValueValidator(-1)])
    created_on_db = models.DateTimeField(primary_key=True, auto_now_add=True)
    updated_on_db = models.DateTimeField(auto_now=True)
    txlist_db = models.CharField(max_length=64, null=False, blank=False)

I solved the problem by changing txlist_db to the primary key.

Turns out each time the bitcoin network tells your server that you've just received some BTC you actually get the signal about half a dozen times in very very quick succession, so fast in fact that even though I had this model in place -

class bitcoin_deposit_txlist(models.Model):
    txlist_db = models.CharField(max_length=64, null=False, blank=False)

and this as this first line of code in the script with the txlist variable representing the network transaction id -

txlist = ''btc network transaction id''
if bitcoin_deposit_txlist.objects.filter(txlist_db=txlist).exists():
    exit()
else:
    bitcoin_deposit_txlist.objects.create(txlist_db=txlist)

I was still managing to get the same record recorded more than once, amazingly.

Here are the values of some of the deposit objects that helped me solve it -

[
{'return_address_db': u'mpfUAfK8bHi4VejvV4WhjEar3yQvPoX1Yn', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 3, 478489, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 12, 30, 804646, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.14384551'), 'confirmations_db': 1, 'txlist_db': u'7fd82945b5b96ae4ff866b6689867038289b81ee947c8796007d3a9f1fa14b59', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mmgC3dk3JRpCrS9UDS8QRHFokFd6N8bLz4', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 3, 491573, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 44, 784596, tzinfo=<UTC>), 'deposit_amount_db': Decimal('1.13722512'), 'confirmations_db': 4, 'txlist_db': u'3f9ad1a2464ca33513f24f0e59274845b4ea04906ab0df054ef6668160f58cb2', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'motQ8oo7RcgMmXZVCHgmbq5jtkGnxJwa59', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 28, 550850, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 31, 254955, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.42293583'), 'confirmations_db': 3, 'txlist_db': u'b8e7e819c0ab1dbae9aa106c3662e17eb3621749a4f72e8e8f1b991212f05971', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'motQ8oo7RcgMmXZVCHgmbq5jtkGnxJwa59', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 28, 559947, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 31, 165585, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.42293583'), 'confirmations_db': 3, 'txlist_db': u'b8e7e819c0ab1dbae9aa106c3662e17eb3621749a4f72e8e8f1b991212f05971', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 666174, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 44, 837193, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 4, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 678403, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 31, 293935, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 3, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 688175, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 15, 53, 125196, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 2, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 695733, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 44, 611634, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 4, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}
]

Notice how close the created_on_db field values are.

Thanks again :)

Upvotes: 0

se7entyse7en
se7entyse7en

Reputation: 4294

Try this:

db_filtered = model.objects.filter(input_address_db=i.input_address_db).distinct()

Anyway by iterating db_objects as you do, you may iterate over the same input_address_db in more than one iteration. I would do something like:

input_addess_dbs = set(d.input_address_db for d in model.objects.all())
for address in input_address_dbs:
    db_filtered = model.objects.filter(input_address_db=address).distinct()
    ...

Upvotes: 4

Related Questions