Same queryset evaluation code, different field names in Django Views

Question

I have seemingly redundant code in Django Views which evaluates the same query-set (Model.objects.all()) but with different field names:

def overview_view(request):
bacteria = Bacteria.objects.all()
bacteria_count = bacteria.count()

bacteriaFilter = BacteriaFilter(request.GET, queryset=bacteria)
bacteria = bacteriaFilter.qs
remaining = bacteria.count()

glucose = []
not_glucose = []
pure_glucose = []
for bug in bacteria:
    if (bug.glucose_acid_from is not None) and (bug.glucose_acid_from != 'neg'):
        glucose.append(bug.species)
for bug in bacteria:
    if (bug.glucose_use is not None) and (bug.glucose_use != 'neg'):
        glucose.append(bug.species)
any_glucose = len(set(glucose))
for bug in bacteria:
    if (bug.glucose_use is not None) and (bug.glucose_use == 'neg') and (bug.glucose_acid_from is not None) and (bug.glucose_acid_from == 'neg'):
        not_glucose.append(bug.species)
no_glucose = len(set(not_glucose))
for bug in bacteria:
    if (bug.glucose_use is not None) and (bug.glucose_use == '+'):
        pure_glucose.append(bug.species)
for bug in bacteria:
    if (bug.glucose_acid_from is not None) and (bug.glucose_acid_from == '+'):
        pure_glucose.append(bug.species)
mix_glucose = any_glucose - len(set(pure_glucose))
nonr_glucose = bacteria_count - any_glucose

fructose = []
not_fructose = []
pure_fructose = []
for bug in bacteria:
    if (bug.fructose_acid_from is not None) and (bug.fructose_acid_from != 'neg'):
        fructose.append(bug.species)
for bug in bacteria:
    if (bug.fructose_use is not None) and (bug.fructose_use != 'neg'):
        fructose.append(bug.species)
any_fructose = len(set(fructose))
for bug in bacteria:
    if (bug.fructose_use is not None) and (bug.fructose_use == 'neg') and (bug.fructose_acid_from is not None) and (bug.fructose_acid_from == 'neg'):
        not_fructose.append(bug.species)
no_fructose = len(set(not_fructose))
for bug in bacteria:
    if (bug.fructose_use is not None) and (bug.fructose_use == '+'):
        pure_fructose.append(bug.species)
for bug in bacteria:
    if (bug.fructose_acid_from is not None) and (bug.fructose_acid_from == '+'):
        pure_fructose.append(bug.species)
mix_fructose = any_fructose - len(set(pure_fructose))
nonr_fructose = bacteria_count - any_fructose ...etc.

I'm using the data from this View to populate a table in an html page:

<div>

<hr />
<table class="table table-sortable table-bordered table-hover">
    <thead>
        <p>Bacteria utilisation or oxidation/fermentation (O/F) of selected carbohydrates</p>
        <tr>
            <th>total</th>
            <th>Carbohydrate</th>
            <th>Any use or O/F</th>
            <th>Mixed response (w, d, vr)</th>
            <th>Neg for both</th>
            <th>Neg or not reported</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>{{bacteria_count}}</td>
            <td>glucose</td>
            <td>{{any_glucose}}</td>
            <td>{{mix_glucose}}</td>
            <td>{{no_glucose}}</td>
            <td>{{nonr_glucose}}</td>
        </tr>
        <tr>
            <td>{{bacteria_count}}</td>
            <td>fructose</td>
            <td>{{any_fructose}}</td>
            <td>{{mix_fructose}}</td>
            <td>{{no_fructose}}</td>
            <td>{{nonr_fructose}}</td>
        </tr> ... etc.

This gives the desired table:

enter image description here

However, I'd like to ask:

1. Is there a large cost to doing multiple evaluations of the data from a remote database? My understanding is that the bacteria = Bacteria.objects.all() query set is cached in memory and can be re-evaluated by subsequent queries. Does this apply here?
2. The code is awfully repetitive, where the only thing changing is the field names. Is there a better way of organising this code?
3. Ultimately, I'd like to allow a custom user input for the different field names rather than hardcoding them in. Is it possible to replace dot notation fields on the fly? Thank you.

Answer 1

Filtering an existing queryset does make an additional call to the database in this case, because the first queryset has been evaluated via .count(). However, count() is more efficient than a full query, so the overhead isn't that great. Unless you are hurting for efficiency, this is probably OK.

However, it is somewhat inefficient to cycle through the recordset multiple times when you are using the same looped items and often testing for the same things. the below only loops through once, and groups some If statements which makes it easier to see patterns:

def overview_view(request): bacteria = Bacteria.objects.all() bacteria_count = bacteria.count()

bacteriaFilter = BacteriaFilter(request.GET, queryset=bacteria)
bacteria = bacteriaFilter.qs
remaining = bacteria.count()

glucose = []
not_glucose = []
pure_glucose = []
fructose = []
not_fructose = []
pure_fructose = []

for bug in bacteria:
    if (bug.glucose_acid_from is not None):
        if (bug.glucose_acid_from != 'neg'):
            glucose.append(bug.species)

        if (bug.glucose_use is not None) and (bug.glucose_use == 'neg')  and (bug.glucose_acid_from == 'neg'):
            not_glucose.append(bug.species)            
        if (bug.glucose_acid_from == '+'):
            pure_glucose.append(bug.species)
        
    if (bug.glucose_use is not None): 
        if (bug.glucose_use != 'neg'):
            glucose.append(bug.species)
        
        if (bug.glucose_use == '+'):
            pure_glucose.append(bug.species)


        
    if (bug.fructose_acid_from is not None): 
        if (bug.fructose_acid_from != 'neg'):
            fructose.append(bug.species)
        
        if (bug.fructose_use is not None) and (bug.fructose_use == 'neg') and (bug.fructose_acid_from == 'neg'):
            not_fructose.append(bug.species)
        
        if (bug.fructose_acid_from == '+'):
            pure_fructose.append(bug.species)               

    if (bug.fructose_use is not None):
         if (bug.fructose_use != 'neg'):
            fructose.append(bug.species)     
            
        if  (bug.fructose_use == '+'):
            pure_fructose.append(bug.species)


        
any_glucose = len(set(glucose))        
no_glucose = len(set(not_glucose))
mix_glucose = any_glucose - len(set(pure_glucose))
nonr_glucose = bacteria_count - any_glucose        
any_fructose = len(set(fructose))  
no_fructose = len(set(not_fructose))      
mix_fructose = any_fructose - len(set(pure_fructose))
nonr_fructose = bacteria_count - any_fructose ...etc.

For dynamic fieldnames, you might try:

str_variable = "glucose_acid_from"
bacteria = Bacteria.objects.all.values()
for bug in bacteria:
    if bug[str_variable] is not None: