The road to RFC. Or elsewhere?

Spec
1

Hi everyone,

As you all know, we’re currently publishing the specification drafts for the remoteStorage protocol as IETF Internet Drafts. Many thanks to @michielbdejong and @fkooman for all their efforts and having been editors for so many years! :heart:

Seeing as we just came up on the 10th version, and hence the 5th birthday of the IETF drafts (after having published in a W3C Community Group before then), and also seeing as recent versions incorporated either very small changes, or even no changes at all, I think it’s time to discuss when, where, and how we want to move towards RS becoming a standard.

There are of course pros and cons to the various options and standards bodies for publishing such a final specification, and the very question if it should be final, and/or if it should be extensible by add-on specs, or completely replaced by a new version later on. Also, as a grassroots movement, we’re not required to publish with any standards body whatsoever, of course.

The IETF route

The draft currently states that its intended status is a Proposed Standard. Here’s a good starting point for learning about the IETF standards process, and I can also highly recommend reading The Tao of IETF: A Novice’s Guide to the Internet Engineering Task Force, which I got linked to by @michielbdejong a while back, and after which I felt that I had a decent grasp on what exactly the IETF is and is not.

The benefits I see to the IETF route:

  • RS is based almost entirely on IETF RFCs. The people in charge of giving feedback and deciding on its progress towards RFC (i.e. the IESG) are best equipped to give feedback on it and ask the hard questions imo.
  • No influence, or even voting on it, by corporate stakeholders
  • We already tick a lot of the boxes on their requirements list, i.e. multiple working implementations
  • If the draft gets rejected, at least we have a better idea of where we stand and if it’s still worth publishing it with a standards body

The major drawback—and I don’t really see another one—is that we have to cede any and all control over the spec to the IESG, who can actually edit, add and remove anything they like. However, as we’re not going there with a half-baked idea, but something we actually use daily already, and have iterated on for 5 years, I think even if they do, it’s unlikely we’d end up with some FrankenStorage spec afterwards.

Missing features

I had a nice, long chat about the spec, and especially about missing features, with @untitaker at FOSDEM recently. We both agreed that the partial/resumable upload topic is the one thing we wouldn’t want to miss in a final version of the spec. Other than that, I don’t see any big features people are still asking for in the current issue list: https://github.com/remotestorage/spec/issues/

Not withstanding a decision on where we should take RS next, I think after having published a spec with no changes last December, it’s time for anyone still missing features or having any issues with the spec as is, to document those, either on GitHub or here.

WDYT?

So, what do you all think? Is 2018 the year we finally propose RS as an Internet Standard? Or is it too early? Or should we go another route entirely?

1 Like
2

sounds good! As Basti mentioned that he and @untitaker still want to make some changes/additions, maybe we can resolve any issues that people still want to resolve in one more Internet Draft?

And then contact the IETF to see what the process would be?

3

Great!

Regarding the uploads: I was merely 1) doing issue triage (and btw there’s an open PR from me to review on GitHub) and 2) talking about it with @untitaker from the perspective of RS users. He’s currently not interested in editing the spec, and from the issue it looks like other people already know much more about that topic than I do.

I’m glad to participate and contribute to that topic. Just making sure that nobody thinks that there are two people who announced that they want to make these changes.

4

To me the biggest problem apart from lack of partial uploads and the resulting performance problems is the missing tooling around the standard. Some ORM around remotestorage would be nice. @raucao said that @michielbdejong had something like that in an experimental JS client already, and that Ember has something like that too IIRC

I don’t see the harm in submitting it as IETF RFC, although it might be wise to have a plan for forking remotestorage back from the IETF if necessary. I’m not sure how to deal with the branding if it would come to this, and the potential confusion about having two standards with the same name.

5

I think it’s important to note that RS as of now is not a standard, but a draft for a proposed standard.

We actually have more code already than most RFCs had at this stage, and we already use RS in production, which IETF also doesn’t recommend for draft specifications. I think we all agree that there’s really no better way than doing that, if you want the protocol to be useful in the real world. However, I started this discussion in part to acknowledge the current draft status and to discuss how to progress it to becoming first a finished, peer-reviewed proposed standard, and later an actual standard by way of broad implementation.

I think this is about a comment I made about search/indexing in the old contacts module. My opinion on this is that those things can and should now be developed as utility modules, which can then be used by any data modules that need the functionality. Our migration to using npm for everything made this possible for anyone to do now.

There were also conversations about a more generic approach to managing data via rs.js regularly in the past. For example forgoing the entire structure of a data module, and only have schemas and e.g. lists (for subdirectories) defined. I think this can also be done using utility modules and just not having much code at all in the data module itself, but importing whatever is needed and just using a nice DSL for describing the functionality. I believe that keeping these things seperate from rs.js core just makes it much easier to both experiment with and later maintain them.

In any case, as you seem to be saying as well, that conversation is fairly irrelevant for the underlying protocol and its submission to a standards body. Which does not make it less important, of course.

Not sure what such a plan could look like and how viable it would be. Would love to hear some more opinions on this!

6

As always, I’m late to the party :slight_smile:
I miss two big features in RS (a part from resumable upload/download, that I will address with something like resumable.js as already discussed in this issue):

Team / Group (see this old issue)

I think this must be addressed somehow.

  • I see very few use cases to manage conflicts (via ETag) without collaborative working.
  • This will open lots of new possibilities in RS usage.
  • Using oauth scope and different parent path for team resources should be pretty easy to implement
    (users to groups map and groups to resources map, ACL would be a plus).
  • As a fallback, I propose to think to an easy way to share entire folders
    from one account to another (this could be addressed outside the RS spec of course).

Content encryption

I think nobody will store his data to a (not self-hosted) RS provider in plain-text,
and in case of self-hosted RS server would be a good feature anyway.
I don’t know if this should be addressed in the spec or not, but I think this should be a
default and each module could use it transparently (think about lesspass).

How to implement it? I propose to use the OpenPGP standard (RFC 4880), store private key
client side only, propose local backup to the user, encrypt it simmetrically with user password as
standard). This could be resolved enterely in client-side (so there should be no reason to touch the
spec) except the case of collaborative resource, that will only be possible by sharing the public keys
of each user to RS server.
I’m aware this will be a big change, but not so big, server-side we will only need to store another
user’s field with the public key.

If you’re interested or at least not completely versus these ideas, I would start
a deeper reasoning on them :slight_smile:

7

I think this is already addressed (kind of), however it’s a bit hidden: a server can issue tokens for shared addresses to users of an organization, and it can even decide to only issue tokens according to permissions. We could mention this in the spec, but I’d prefer to have used this myself before documenting it.

The other, more obvious option is using long URLs in /public as authorization.

These two options are really powerful and under-explored imo. And personally I think we should leave the very complicated ACL/collab stuff to SOLID, while keeping RS nice and lean. I always felt that the beauty of RS is that it’s so simple and only a very small layer on top of pure HTTP, so it’s easy to reason about and implement for anyone.

As you say, this basically needs zero changes in the spec. As you can store whatever you want, this is entirely a client problem to solve. I would agree that this is an important one as an option for developers, but I would disagree that it makes sense for every use case and user. We should probably put together a roadmap for rs.js, so people both know what the plans are, and can also start working on these things.

If you want to address this, please contact @silverbucket. He just started looking into the uploads last weekend! Got a lot of experience with client-side JS data transfer (as he’s working for Akamai) as well as node.js (author of Sockethub, Jaribu, webfinger.js, among others). And he actually planned to prototype this in Armadietto!

8

It sounds interesting to me, but where group’s information should be stored?
Thinking about authentication / permission request, why would the widget is asking for the password?
wouldn’t be better to enterely rely on oauth page, so we can split the whole funcionality?

I’m glad to hear this, I’ll help when/where possible :wink:

9

I’m not sure I understand your question. The idea of the storage provider handling this via login/tokens means that it is entirely seperate from the widget (and any of the current spec). You would give it the user address of your group/organization/team, and then the provider could give you a token for that, if your own user account on that server has permission to access the org’s account. So you would use your normal login on the server, but see the OAuth and receive a token for the group account.

10

I meant (a part from group issue), I don’t know any other places where a service is asking for your password to authenticate to a third service:
for instance, take this forum based on discourse, when I want to authenticate myself via github, the form asking for my password is the github oauth page, not a login page in this same forum (using it that way I need to authenticate only once for all website where I’m using my github identity). So my specific question was, why is the widget asking for the password and not the oauth page of the RS server? It makes sense?

About group/team issue, what about just introducing a new basepath? (like how it is with /public, let’s say /group), I think this should be a reasonable modification.

ps. sorry for the long delay :pensive:

11

I think I lost the thread there. When and where would the widget be asking for a password? With the OAuth redirect flow that RS is using, this should never ever be the case.

12

:confused: OMG You didn’t lost anything, I’ve lost my mind! I confirm the widget is not asking password anywhere. I’m sorry for the confusion.

I confirm the group/team proposal desire, I’ll try to make some experiment.

13

By the way, aaronpk just published a blog post about IndieAuth: https://aaronparecki.com/2018/07/07/7/oauth-for-the-open-web

Good to see that they arrived at exactly the same conclusions as we did with RS OAuth, regarding client ID, redirect URIs, etc.

14

Now that we’re in the process of implementing multi-part uploads, I was thinking about that a bit more, and this specific question stuck in my head:

Going back to first principles, the reason why we have to design the upload protocol in the first place is that there’s no HTTP RFC for this problem. But if that’s the case, why solve this problem only for ourselves, when the idea of RS is to just use a subset of other open protocols, with minimal sprinkling on top to tie everything together.

So, the question is: wouldn’t it make more sense to submit this as a standalone multi-part upload protocol and only reference it as an optional add-on for servers and clients (using our Webfinger add-on discovery) in the RS spec?

Same goes for the implementation then, of course. We could make the pure upload functionality its own library that any other client, not just RS ones, could import and use.

1 Like
15

:point_right: Public RS Planning/Community Call (April 21, 6pm CEST / 9am PDT)

1 Like
16

One thing IETC might ask to change that I forgot about in the call is the OAuth grant type. Draft 14 uses Implicit Grant, but it is considered deprecated in favor of Authorization Code + PKCE.

1 Like
17

Yes, agree! I also started an issue about that a while ago, see https://github.com/remotestorage/spec/issues/176

1 Like
18

Dropbox also introduced PKCE recently, but continues to support implicit flow with short-lived tokens:

I think that’s a good compromise for backwards compatibility. Also, if we decide to add PKCE to RS, then we can use the same client implementation in rs.js for both RS and Dropbox now.