Paul Chernoch
Paul Chernoch

Reputation: 5553

Can Voila cache data to be reused when a widget changes value?

I am loading large files with 10-100 million 3D points and multiple labels per point into a Pandas dataframe. These points are then filtered into a thin planar slice along a chosen axis and projected onto a plane, with color based on the label chosen. For large files the rendering is slow, because every time you select new filter values in the widgets, it causes Jupyter to reload the file, refilter the points, and regenerate the plot.

Is there a way to have Voila/Jupyter Notebooks cache the data loaded from the file? I intend to sort it by each point’s distance from the plane of the slice, then use binary search to find and filter the points to be projected. If the user changes the axis they are stepping through with a widget (like a slider or numeric field), I will resort the cached data to reindex the points. If they switch to a different file, I will clear the cache.

Is this possible? My fallback is to load data, compute the distance from the plane of slices, add this distance column to the dataframe and save it to a file. I save on sorting but lose on file IO.

My solution is constrained because everything is on a secure server so I can’t create a Redis cache or similar thing to offload the caching to another component.

Upvotes: 1

Views: 113

Answers (0)

Related Questions