I’ve been working on a Javascript-based web filesystem for a few years called JSFS (https://github.com/jjg/jsfs), primarily geared toward storing and delivering large media files (hence the focus on deduplication, etc.) I ran across remoteStorage by way of Laverna (https://laverna.cc) and after reading the RFC was excited to see just how similar the API is to where I’ve ended up with JSFS. I’ve been a fan of the “Unhosted” web app architecture for awhile (which I’m sure has influenced the design of JSFS) so I was very happy to see a project like this underway.
Based on this, as well as the fact that a lot of the objectives I had when I set out to build JSFS overlap with remoteStorage, I’m going to take a shot at integrating remoteStorage support into JSFS. I don’t expect this to take require any radical changes but there are some subtle things that might involve breaking changes on the JSFS side but I think it’s worth it to maintain compatibility.
One question I have off-the-bat, is there a good “validation” test that I can use (preferably something automated) to run against my changes to make it easier to know when I’ve covered the requirements for openStorage compatibility?
Thanks for your work on this project and I look forward to doing what I can to participate!
So will JSFS become a remoteStorage client or a remoteStorage server?
I know we say several times each year that we need a good validator, and many people have started writing one, but I don’t think any of them were ever officially published.
I must admit I haven’t heard of JSFS until now. Sounds very interesting and indeed looks similar. Nice work! The way I understand it is that you want JSFS servers to be compatible with remoteStorage clients, right?
If that’s the case, then regarding the test suite, there’s only an unfinished one by Francois Kooman I think (not sure if he published it). As @michielbdejong said, everybody’s super interested in having a general-purpose test suite, but so far there is none.
Initially as a server, although I’m also looking into making the backing-store more flexible which might make it behaving as a remoteStorage client make sense as well at one point.
Cool! Our getting-started information is a bit hard to find (we have an old project website, but it is a bit abandoned in the last years). The links you need are:
I would add the disclaimer that most of those links are needed for app development, as opposed to server development.
@jjg Would be great if you can open GitHub issues on the spec, in case you stumble upon things that are unclear or that you think can be improved! Here’s the repo: https://github.com/remotestorage/spec
There’s also reStore, the de-facto default remoteStorage server program, writte in node.js, which you can peruse: https://github.com/jcoglan/restore/
We’re also in #remotestorage on Freenode if IRC is convenient for you.
Awesome, I found a few of those things but not all of them, I’ll queue them up for digestion asap
I read the RFC and it was as clear as any other RFC I’ve read I do have a couple questions, but I thought I should work through using it a bit to see if that answered my questions. The one thing that stood out was the lack of the POST verb, which I currently use in JSFS for creating new objects (I currently block requests that attempt to POST over an existing object and block PUT requests that would result in a new one). I can guess a few reasons why POST was omitted but I’d love to know the “official” answer.
Thanks again for this project. Oh one more question: is there a similar effort underway that you are aware of for remote processing? JSFS is a component of a larger project I’m working on which includes a distributed processing architecture designed to provide compute resources to unhosted-style applications. AFAIK there isn’t another implementation or standard that provides exactly what I have in mind, but then again I worked on JSFS for a year or so before finding remoteStorage so I could just be oblivious to similar efforts.
There’s no real reason that we picked PUT instead of POST, except maybe we wanted only 1 verb for both create and update, because it’s a simplification. At one point we even considered using a PUT with empty body to mean DELETE, but when we tried to write that down we realized it actually caused more confusion than simplicity.
I do know POST is sometimes used in APIs to let the server choose the URL of the item that gets created (for instance POST to api.com/users/ to create api.com/users/156257 if that’s the next user id). Originally, POST came from html form posting, the analogy of filling out a form with pen on paper, putting it in an envelop, and sending it in the (snailmail) post. Whereas PUT comes from the analogy of “I’m putting this documented onto the server”. This seems to be confirmed by http://tools.ietf.org/html/rfc7231#section-4.3.3 and http://tools.ietf.org/html/rfc7231#section-4.3.4 (just looked that up ).
Just looked at JSFS, cool stuff! So would you say it defines both the API as well as the implementation? I’m still a bit puzzled by the keyring part, I understand I need either an access key or access token to do something, but how do I obtain those?
Re protocols for generic sandboxed computations - I think that problem is a bit too generic to put into a one-size-fits-all API. There are many projects that try to decentralize computing, either into peoples browsers, into people’s unused computers during the night, and yesterday I saw one for running computations to heat other people’s houses
Usually the way to actually run the code would be in a VPS with linux and an IPv4 address, so the API is basically bash-over-ssh.
If it wasn’t considered back then, here’s another reason for PUT: it MUST be idempotent, whereas POST does not. For a key/value store (which RS servers are in the end), updating a key, no matter if it existed before or not, should obviously be idempotent, i.e. not create arbitrary new resources on create requests. Also, for a key/value store it doesn’t matter if a resource has been created before, as long as there’s no conflict when trying to update it.
So PUT can be used for both creates and updates in the same way (which I think is what @michielbdejong means by simplification) , and the server may respond to creates with 201, so the client knows what happened.
I’ll check that out @raucao. If nothing else I’ll have to write my own test harness to make sure I’ve done it right so if I go that route I’ll try to make it something reusable
You are correct that my plan is to see if I can make JSFS compatible with remoteStorage clients. Having read over the RFC I think that the two are pretty close as-is, but there are some details that are not quite the same. That said, I think that adopting the remoteStorage way for some of these will make JSFS better as well (the address things I didn’t have great solutions for myself). There are some things that JSFS does that might not make sense for remoteStorage but I think they will just be ignored by remoteStorage clients.
@michielbdejong I can totally see using PUT for simplicity and idempotentcy (is that a word? :). I opted to use POST and PUT to make it unambigious to the client if it is creating something vs. updating an existing resource but @raucao’s mention of using 201 solves that problem and maintains simplicity which I think makes client application developers lives easier so I like it.
At least for now I think I’ll keep POST in the JSFS API but support using PUT the way its used in remoteStorage, that way it can be remoteStorage compatible but also backward-compatible with existing JSFS clients (it also helps me prevent having long boring ideological conversations with REST purists
@michielbdejong regarding keyrings and such, that was my solution to handling authorization for shared resources. JSFS doesn’t implement an authorization scheme directly because my initial focus was on creating a storage server and I didn’t want to get bogged-down in coming up with a good authentication/authorization system, so I just sort of “outsourced” that problem to the client This is an area where I want to learn how remoteStorage handles these things and potentially adopt that model as well.
Right now every object stored in JSFS has an access key that can either be generated by the system (default) or supplied by the client when the request is made. This gives the “owner” of the object (anyone possessing the key) absolute control over the object. The token system was devised to allow an owner to delegate partial control to other clients as is needed for shared objects. This is done deterministically so tokens can be generated by the client instead of having to request them from the server. A side-effect of this is that an owner can immediately revoke all tokens by changing an objects key.
Using this mechanism, we came up with the pattern of a “keyring” to emulate an authentication system (essentially a deterministic way of unlocking another object that contains a user’s keys), but again this wasn’t really central to JSFS’s design, and another area where I hope to learn something from how remoteStorage handles authorization.
True. I wasn’t talking about surprises and protecting data, though, but just that you often want different UI for creates vs updates, even if it’s just a flash message or different icon somewhere.
Anyway, this is a very good point, and I think we need to document these concepts in a concise manner somewhere for people who want to understand the protocol. Maybe in a simple FAQ for it? “How does remoteStorage solve conflicts when connecting from 2 clients?” “…”
P.S.: Incidentally, this is the kind of content, which we wanted to set up the wiki for. Especially so it’s easy to add and edit, even more so for “drive-by” improvements, where people don’t want to spend time on learning how a website build system works, or having to start a PR in an official repo for it.
@michielbdejong I’ll read up on that (OAuth implicit grant flow). I’ll admit I’m pretty ignorant to OAuth, I tried to get into it awhile back but it was a non-starter because it seemed like it required a browser to work but that may have been my misunderstanding (or maybe it has changed since then).
Since a lot of the systems I build are autonomous, having a step that requires a web browser was a non-starter.
Just a quick update: we’re almost done with the test suite (in fact we used it to add an OpenStack Swift backend to Liquor Cabinet), but it’s missing some specs for permissions and public access, as well as a nicer README.
I do have a couple questions, but I thought I should work through using it a bit to see if that answered my questions. The one thing that stood out was the lack of the POST verb, which I currently use in JSFS for creating new objects (I currently block requests that attempt to POST over an existing object and block PUT requests that would result in a new one). I can guess a few reasons why POST was omitted but I’d love to know the “official” answer.
).