this week i tried to store 5000 email headers in an unhosted web app; about 8Mb of data when JSON-stringified. when you compare it to standard lamp or native C/C++ software development (where anything under 1 million rows is considered a small dataset), the experience was really quite slow and problematic.
we also already found the problem that doing many consecutive commits to IndexedDB is slow (importing an addressbook into remoteStorage takes several minutes), and sometimes IndexedDB just throws an error at times where you don’t want it to.
basically, i think we should move IndexedDB out of the critical path of how an app accesses its data.
i think the way forward is to start using an approach more like memory paging. this would require a change in the baseclient API:
- keep subtrees of the data tree in memory, and allow synchronous access to that.
- provide a function to load and unload certain subtrees from IndexedDB to keep mem usage down
- the official local copy of the data is in memory, and from there we push a changes feed both to IndexedDB and to remote (if connected).
- for efficiency, data in IndexedDB can be stored in bigger composite objects, representing whole subtrees. we can also store incrementals, and replay them on top of the latest snapshot when recovering from a browser crash or page refresh. these incrementals can then be applied to the snapshot when the subtree is unloaded from memory.