CODEWITHSUNDEEP
databaseapache-superset
Bad Coder
Bad Coder

Reputation: 906

Apache SuperSet Architecture

I bit confused about how Superset process query. As per my understating Superset only store metadata i.e users, dashboard definition. It will not store the query data ?

If my query is returning let's say 10GB of data how superset process it ?

Upvotes: 0

Views: 2013

Answers (2)

Daniel Harapko
Daniel Harapko

Reputation: 21

I think this should be resolved through an additional cache database.

"Superset uses Flask-Cache for caching purpose. For security reasons, there are two separate cache configs for Superset's own metadata (CACHE_CONFIG) and charting data queried from connected datasources (DATA_CACHE_CONFIG). However, Query results from SQL Lab are stored in another backend called RESULTS_BACKEND. Configuring caching is as easy as providing CACHE_CONFIG and DATA_CACHE_CONFIG in your superset_config.py that complies with the Flask-Cache specifications."

https://superset.apache.org/docs/installation/cache

Your queries can be gigantic when using async queries via celery https://superset.apache.org/docs/installation/async-queries-celery

Upvotes: 2

grace dmello
grace dmello

Reputation: 17

Superset has meta-database which stores configuration information. For example, dashboard and chart configuration information.

The actual response from the exploration database is never saved.

Regarding the size limitation of the result please refer https://superset.incubator.apache.org/faq.html#how-big-can-my-data-source-be

Upvotes: 0

Related Questions