Keep Upload Data for Analyses in Flask Session

Question

I am very much struggling with a concept and hope someone can offer a code implementation suggestion. I have the following route embedded within my flask app. After working on this for a few days I think I now see the nature of the issue. I have modified my code based on suggestion below in comments.

In the web app now, the names of the data frame columns populate a drop down menu and the user can choose one of those variables and click "Show". That variable name then prints to the screen just so I know my POST from the form is communicating with my function in the flask app.

What I would like to do is create a temp version of the file uploaded so that it exists as long as the web session is open (or is overwritten if a new file upload occurs). Then, the user chooses a variable from the drop down menu and the mean of that variable is computed (that part is commented out in the code below for now but you can see what I have tried).

If the user reads in a new file, then the same process would occur on that new file.

Thanks for any suggestions or supports.

from flask import Flask, render_template, request
import numpy as np
from scipy.stats import binom  
from scipy.optimize import minimize
from scipy.stats import norm 
from scipy import optimize 
from pyodbc import connect 
import pandas as pd 
import os
import tempfile


app = Flask(__name__)
    @app.route('/dataTools', methods=['POST', 'GET'])
def data_tools_upload():
    tempfile_path = tempfile.NamedTemporaryFile().name
    if request.files:
        df = pd.read_csv(request.files.get('file'))
        #tempfile_path = tempfile.NamedTemporaryFile().name 
        df.to_csv(tempfile_path)
    if os.path.exists(tempfile_path):
        orig_df = pd.read_csv(tempfile_path)
        vars = list(orig_df.columns)
        var2use = request.form.get("var2use")
        #indx = vars.index(var2use)
        #df[vars[indx]].mean()
        mean = orig_df[vars[4]].mean()
        dims = orig_df.shape
        message = 'You have data! There are %s rows and %s columns and the variable %s has mean %s' % (dims[0],dims[1],vars[4],round(mean,3))
        table = orig_df.head(10).to_html(classes='data', header = "true")
        return render_template('upload.html', tables = [table], message = message, vars = vars, var_name = var2use)
    var2use = request.form.get("var2use")
    return render_template('upload.html', var_name = var2use, message = "What happened?") 
if __name__ == '__main__':
    app.run(debug=True)

And then the following HTML bit (upload.html):

{% extends "layout.html" %}    
{% block body %}

<h3> Read in a data file </h3>

<br>

<form method=post enctype=multipart/form-data>
    <input type=file name=file class = "btn btn-outline-secondary">
    <input type=submit value=Upload class = "btn btn-outline-secondary">
</form>

<br>    

<form class="form-inline" action = "{{url_for('data_tools_upload')}}" method = "POST">
    <select type = "text" name="var2use" class="custom-select mr-sm-1">
        {% for var in vars %}
            <option value= "{{ var }}" SELECTED>{{ var }}</option>"
        {% endfor %}
    </select>
        <button class = "btn btn-primary"> Show </button>
</form>


<center>
    <h1>
    {{message}}
    {{name}}
    </h1>
    <br>
        <small>
        {% for table in tables %}
                    {{ table|safe }}
        {% endfor %}

        {% endblock %}
        </small>
</center>

Answer 1

you need to save file to work with it (the code works perfectly in my app):

        f = request.files['file']
        tmp_filename = '/tmp/my_tmp_file.csv'
        f.save(tmp_filename)
        pd.read_csv(tmp_filename)

Here is the detailed documentation on Flask file uploads: https://flask.palletsprojects.com/en/1.1.x/patterns/fileuploads/

Full example for your case is below. I think you might have problems with the following three items:

1. Uploading the file because of form enctype (should be "multipart/form-data")
2. Uploading because of file size (try my sample)
3. Processing the form with empty file overwriting yours (see my example for "csv_file_uploaded")

Python (csv_test.py):

from flask import Flask, render_template, request
import pandas as pd
import os

app = Flask(__name__, static_url_path='')

@app.route('/', methods=['GET', 'POST'])
def index():
    data = {}
    tmp_filename = '/tmp/csv_test.csv'
    if request.files:
        csv_file_uploaded = request.files.get('file')
        if csv_file_uploaded:
            f = request.files['file']
            f.save(tmp_filename)
    if os.path.exists(tmp_filename):
        df = pd.read_csv(tmp_filename)
        data['columns'] = ', '.join(df.columns)
        col_selected = request.form.get('col_selected')
        print(col_selected)
        data['col_selected'] = col_selected
        if col_selected:
            mean = df.get(col_selected).mean()
            data['mean'] = mean
    return render_template('csv_test.tpl', data=data)

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

Flask template (csv_test.tpl):

<html><head><title>CSV TEST</title></head>
    <body><h1>CSV TEST</h1>
      {% if data %}
          <div>Columns of the uploaded file: <strong>{{ data.columns }}</strong></div>
          <div>Selected column: <strong>{{ data.col_selected }}</strong></div>
          <div>Mean of the selected = <strong>{{ data.mean }}</strong></div>
      {% endif %}
      <fieldset>
         <form enctype="multipart/form-data" action="" method="post">
            <h2>Provide some CSV with first column in numbers and/or column name</h2>
            <div>File{% if data.columns %} (UPLOADED){% endif %}: <input id="file" type="file" name="file"/></div>
            <div>Column: <input id="col_selected" type="text" name="col_selected" value="{{ data.col_selected }}"/></div>
            <div><input id="submit" type="submit" value="SUBMIT"/></div>
         </form>
      </fieldset>
</body></html>

CSV file (123.csv) - use it to test the form:

col1,col2
1,2
2,3
3,4
5,6