CODEWITHSUNDEEP
javascriptchromiumindexeddb
jcc
jcc

Reputation: 1147

Allocate browser memory for IndexedDB connection open issues

The Setup:

Hello, my org has built an Angular PWA for our users to work offline in the field on their corporate Surface Go devices. This PWA needs to review and edit various data while users are disconnected, so we researched and settled on working with the IndexedDB (Idb) in the browser. Users have the ability to download various files, PDFs, and images while in the office, go offline and into the field, capture new photos within the app (Idb), and upload this all back to our servers upon return.

We store large JSON and blob objects in the Idb to keep track of everything and users have a good workflow to follow for downloading, fielding, and uploading. We also keep a close watch on the Storage within the browser to ensure the Surfaces do meet the limits.

The PWA is loaded in MS Edge (Version 96.0.1054.62 (Official build) (64-bit)) on Windows and the PWA is installed onto their desktop to function like a native app. The Surfaces have 8GB of RAM and plenty of hard drive space for storage.

The Problem:

This has all worked great for a few months but users have now started experiencing issues with the PWA crashing upon navigating to our app. The browser is opened, navigate to the URL, the app loads for 2-3 seconds, and then the entire browser crashes (all tabs and windows). Users have reported uploading and storing photos (3MB each) when the first crash happens. Users are then out of commission and can no longer access the app by any means, so there defiantly seems like a tipping point somewhere.

We have narrowed it down to a memory (RAM) issue with the browser. When the app starts up and establishes a connection to the Idb, the memory spikes dramatically based on the amount of data stored within it. Looking at the local File System in Windows the users that have been affected have upwards of 4.5GB+ stored in their Idb (this is everything).

Browser Memory Spike

Loading the browser .exe into Visual Studio and navigating to the app we can see the result. As soon as the call the indexeddb.open happens, a spike in memory is shown. Chrome memory spike

Chrome (in this instance) throws memory errors.

We have also tried to increase the amount of memory available to the browser by following these posts with no difference. We have been able to copy out the Idb files from the affected users (C:\Users{user}\AppData\Local\Microsoft\Edge\User Data\Default\IndexedDB), swapped them into our machines, and been able to reproduce the crashing locally.

enter image description here

System memory shows the same spike, but there is obviously a lot more memory to give.

As it stands, our users are dead in the water and unable to open the app and unable to access their data. The raw Idb files are unreadable and every solution we have taken to extract the data (https://github.com/amyboyd/indexeddb-to-json) just fails in the same way.

Questions:

  1. Can we allocate more memory to the browser through some other means?
  2. Can we extract the data from the Idb without the browser?
  3. Can we avoid the memory issues when opening the Idb some other way?

Upvotes: 1

Views: 1620

Answers (1)

jcc
jcc

Reputation: 1147

OK, so months later we were able to implement a different approach that goes around the IndexedDB and uses the File System API. This has its own drawbacks but this fixed all my issues by getting raw files out of the IndexedDB.

Related to the IndexedDB itself, we found some very specific issues around the images we were storing there, as they were all being base64 encoded and stored as strings. It seems that when the IndexedDB stores these large strings (3-4MB) it has to decode them when it wakes up to correctly render them on demand at a later time.

Based on the number of large strings it has to do this for when making the initial connection to the database, it causes a huge memory spike (decoding hundreds of base64 strings) and ultimately crashes the browser as the IndexedDB and browser engine just completely crashes.

We were not able to dig too deep into how and what the IndexedDB does under the covers as some proprietary magic from the creators at Google? I'm hoping someone with more knowledge on standards at the WWWC can chime and provide more data on this. I would love to know how it works under the covers and why we can't access the raw data on the file system.

TLDR; Don't store large base64 encoded strings in the IndexedDB.

Upvotes: 1

Related Questions