Syncing large number of items best practices?

I’m working integrating remoteStorage in two apps at the moment and pretty much everything seems to work great. There just a weird thing where it take a long time to start the app if you are both:

  • subscribing to on change events to load objects (I assume you need to subscribe if you want to sync between multiple devices)
  • and loading a ‘large’ number of items (I don’t think 100 is very large, but let’s use that as a reference)

Here’s a test app for testing out the loading time https://remotestorage-quantity-test.herokuapp.com It seems like there is a delay that grows linearly with the number of items you have - it’s not noticeable with 20 items, but if you’re like me and you have over 200 notes, it takes a good 5-10 seconds sometimes and that’s before new changes come in.

If anyone sees some flaw in my approach or has recommendations please let me know. Super excited to work with this technology :slightly_smiling_face:.

I guess I’m also asking in a way: is there a fuzzy limit to the number of documents or maybe a best practice for quickly loading 1000 items in the same folder while being able to sync?

Good to hear!

You’re right. 100 is not a large number at all.

The key to fast loading times is caching your documents. Ideally all of them. Further improvements can be made by not checking for updates when opening the app, by setting maxAge to false for your fetch method (e.g. myClient.getAll(false)).

I had a quick glance at your app and created 600 docs with it. Caching seems to be all there and load fast in general. But loading/rendering seems to be a bit slow indeed. However, from the outside it’s not obvious why.

I manage a few thousand bookmarks with Webmarks (source code), and it loads within 1 or 2 seconds for me. However, I actually had quite the delay from the mere rendering of the items, before I went and fixed that. Now it’s only rendering a bit more than fits the screen, and then adding more items to the rendered list by way of catching IntersectionObserver events from the end of the list (see code changes here).

Excellent, it’s almost instantaneous now after I made the following changes:

  • start by using the value from getAll instead of waiting for change events
  • set maxAge to false as you pointed out (I didn’t know this existed)
  • and then if you are syncing between devices, ignore change events with origin local and oldValue undefined

Thanks so much for the tips, can’t wait to share some new apps soon!

2 Likes