I have some ideas/thoughts about projects that stray from the protocol, and I’m not sure which way makes sense, should one:
create a separate project to rs from scratch?
fork rs? and then merge or not?
add storage adapters that may ‘misuse’ the protocol-oriented library?
add private hooks in the library for non-protocol behaviour?
I find the first two kind of wasteful as:
my needs are not too far from what’s already built
I would like to use the well-tested public code that already works
I think my approach is complementary to the values of the project and perhaps both advance the same interests.
To give some idea of the use cases I’m imagining:
prefer formats that people already use, more in the direction of files, ideally little to no json, maximise interop
connect a private github repository to store remotely (I might like to work on a rs storage adapter for this anyway)
connect a read-only public github repository, to pull files to the browser
add multiple storages simultaneously
‘publishing to public folder’ replaced by connect specific storages and choose where to publish
more ‘file picker oriented’ than ‘locations determined by developer’
perhaps support access to chrome-only File System Access API, with no sync or browser storage
It seems the library might already be able to do this with storage adapters and perhaps some adjustments. Would that be welcome here?
Also might be useful to separate the project name into three things (which I think should be named differently at some point):
remoteStorage the concept: separation of data from apps, “connect your existing drive” like dropbox/google drive, I promoted this with 0data
remoteStorage the protocol: api to standardize communication between servers and clients
remoteStorage the js library: local-first, anonymous-capable, file storage that syncs with multiple back ends. (perhaps the only open-source project that does this on the internet)
The protocol is the only thing that seems quite specific to this project and community, whereas the other two are quite broadly applicable. I believe we can draw interest, usage, and contributions by enabling those with similar interests (but different opinions about the protocol) to use the library.
Curious to hear what people here think about any of these topics.
AFAICS, almost all of what you mentioned can already be built with the current library and its APIs. The only thing that would be new is a GitHub adapter.
The client, not the library determines storage locations
Built-in caching is optional, a client can disable it and use filesystem APIs instead
You can use storeFile and getFile to store and retrieve files in any format
Regarding the name, we already have remoteStorage for the protocol, and remoteStorage.js (emphasis on .js) for the core-maintained JavaScript client library (which also acts as the main reference/example client).
It’s interesting to think of it as a concept, in addition to being a protocol. However, couldn’t that make it a bit confusing when you then try to explain to someone that the protocol is different from the concept? For example, if we used the term “email” for things that are electronic messages, but not using the email protocols, would that be a good thing for both email and the wider public?
Overall, I think you’re hinting at more than what you laid out here, and/or maybe I’m not understanding it correctly. If the use case isn’t really compatible with RS (the protocol) at all, then I guess just building a different library would make sense. But if it generally might work with an RS server, then I think extending the library for whatever your client is missing could be worth it.
To that end, Finishing up the TypeScript and Mocha porting will make extending the library much easier and safer. However, I think there are some limits to the UX you can provide when you want both full filesystem navigation/storage and browser sync that deals with arbitrary changes on multiple devices. Really depends on the specific use case and details.
Btw, I think you don’t need “private hooks”, but you might want to create an add-on library that simply uses the public APIs, similar to how remotestorage-widget uses nothing but public rs.js APIs.
I think this assumes there’s enough consensus such that the name is understood intuitively as both things, and so order for this to be the case with remoteStorage, the protocol would need that kind of adoption; maybe that storage providers also start referring to their ‘drive/cloud’ offerings as remoteStorage, which I doubt they’d do. But maybe we can look at ‘messages’ and ‘messenger’ referring different platforms/protocols depending on the context? I’m considering a landscape that’s more messy, where there isn’t yet a consensus as with email, or where one might not be foreseeable. Finally, even if the protocol and concept could be understood as the same, the library clearly encapsulates broader concepts about data portability (local-first, anon-capable, support for sync with multiple backends); in my opinion, those are the things that make it a jewel, while at the same time, many people have no idea such a powerful developer tool exists because they think it’s only about the protocol or to be used with rs servers – opportunity there for many more people to take notice.
I think I need to read the spec to understand what the difference might be, but with my current layman approach I only see this diff of excluding this 000_folder-description.json file for non-rs servers so that the directory contents remain as it would on any other filesystem.
Interesting if I can support Chrome’s Filesystem API by simply writing a storage adapter and disable caching.
Any thoughts about connecting multiple storages simultaneously?
Is that referring to a file that rs-backup writes to disk? An RS server generally has to provide directory listings, as well as ETags for folders and files, but how it does that on the filesystem is up to the server implementation. Usually, the listings come from some kind of database, but some servers might opt for metadata files.
Yes! There’s an open issue for it, and @galfert was working on that on the side recently, to be able to use it for a data migration app (i.e. migrate one’s data from one RS account to another):
There’s no PR yet, but it’s definitely something that’s both interesting and possible. It would probably be best to discuss the details in a separate topic here or in the relevant GitHub issues.
Regarding the naming again, “remote storage”, without the lowerCase camelCase, is also already used for the concept sometimes. But I think “cloud storage” is much more prevalent, and most people probably know roughly what it means, even if it’s not accurate for specific RS server deployments or self-hosting.
Also, @michielbdejong said that a lot of people at the last Local First conference were calling their libraries “sync engines”, and I guess rs.js falls somewhere in between a sync engine and a cloud storage library, covering both use cases, but not as good as specialized clients for either.
Ok this means I need to read the spec to understand what exactly is necessary – perhaps I already have the parts I need.
Also thought of some other examples where the protocol/platform/underlying is commonly referred to another name:
activitypub vs mastodon
atproto vs bsky.app
nostr vs the client
git vs github
Many can use the latter without ever knowing about the former, which might seem odd for technical people who are aware of the underlying mechanics, but I think it just means more people can join in – easier onramp.
