CODEWITHSUNDEEP
pythonpandasflaskclickhousedata-stream
Nason Thomas
Nason Thomas

Reputation: 55

Best way to send huge data in streaming mode through Python flask application?

I want to create flask application which returns the huge data but the problems is, my api is crashing when it is ask for huge dataset like 6 million records. I want to make my flask application which gives streaming data, where I will just send 100000 records at a time.

What my application looks like:

import clickhouse_connect
import orjson
from flask import Flask

app = Flask(__name__)


@app.route("/")
def send_data():
    client = clickhouse_connect.get_client(host='XXXX', port=XXXX, username='XXXX', password='XXXXX', database = 'XXXXX')
    df = client.query_df("SELECT * FROM Table_Name")
    #table_df.to_dict(orient='records')
    print(type(df.to_dict(orient='dict')))
    return df.to_dict(orient='dict')

Suggestions are welcome make my application even faster.

Upvotes: 0

Views: 90

Answers (1)

Adam Spaulding
Adam Spaulding

Reputation: 1

Sounds like you are using client-side rendering, which is trying to render all of those records at once to the client. I would recommend initializing server-side rendering on your front end, using for instance, DataTables js, or similar library for displaying the data. DataTables would then make Ajax requests to your server for each draw of the table. While not necessarily having finite control over the number of records returned, it would generally only return the records being visibly displayed on the screen. You would, however, have to handle the logic for pagination, sorting and searching on the server, as these functionalities would no longer be managed client-side.

Upvotes: 0

Related Questions