raucao
April 8, 2024, 7:37am
1
@DougReeder just opened a fairly large pull request for Armadietto, with the following list of features and fixes:
streaming transfer (documents don’t need to fit in memory)
JSON Web Tokens (JWT) so permissions can be validated in memory, without a round trip to a server or even reading a file
S3-compatible storage (AWS S3 allows up to 5 TB)
Bug Fix: correctly handles If-None-Match with ETag
Bug Fix: returns empty listing for nonexistent folder
Implements current spec: draft-dejong-remotestorage-22
based on Express, for much cleaner code & standard extension interface
easier to reason about
stores are standard Express handlers
new kinds of authentication will be straighforward
Middleware exists for almost anything that is worth doing in an Express server
Posting this here in order to ask anyone with node.js experience to consider helping with reviews:
remotestorage:master ← remotestorage:modular
opened 03:40AM - 08 Apr 24 UTC
### Features/Fixes
- streaming transfer (documents don't need to fit in memor… y)
- JSON Web Tokens (JWT) so permissions can be validated in memory, without a round trip to a server or even reading a file
- S3-compatible storage (AWS S3 allows up to 5 TB)
- Bug Fix: correctly handles If-None-Match with ETag
- Bug Fix: returns empty listing for nonexistent folder
- Implements current spec: draft-dejong-remotestorage-22
- based on Express, for much cleaner code & standard extension interface
- easier to reason about
- stores are standard Express handlers
- new kinds of authentication will be straighforward
- Middleware exists for almost anything that is worth doing in an Express server
### Support for other extensions
Allows the extensibility of #88 and #89 and armadietto+lucchetto in a standard way. Bespoke versions can be implemented by copying `appFactory.js` and adding new middleware. It will be much easier to say "yes" to extensions with a limited audience. Middleware for storage quotas, rate limiting and in-app purchase can be in the archive, but don't have to be in the default configuration.
### Notes for evaluating
#### Store
To run the server or test S3-compatible storage, an S3-compatible instance must be running and environment variables set. See `notes/S3-streaming-store.md`. To run automated tests:
```
node node_modules/mocha/bin/mocha.js spec/streaming_handler/S3Handler.spec.js
```
- AWS fully works
- Min.IO is usable, but has a leaky abstraction — the Min.IO web Console only displays the root folder & deleting a user doesn't fully complete.
- OpenIO fails the simultaneous deletion test — it shouldn't be used now, but perhaps some day it will be useable
- Could use testing with Google Cloud Storage, [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces), etc.
- Could use more testing with files larger than 1GB — see the "transfers very large files" test
#### Automated Tests
The original automated tests are still in `spec/armadietto`. They validate that the monolithic server still functions the same.
The automated tests have been copied copied and reworked to allow testing the modular server against the same tests as the monolithic server. The tests starting with `a_` in `spec/armadietto` set up the monolithic server and call the tests in `spec`. The tests starting with `m_` in `spec/modular` set up the modular server and call the tests in `spec`. Thus, the tests validate that the modular server behaves the same as the monolithic server, aside from a few edge cases.
1 Like
Thanks, @raucao ! I tried to post this morning, but the forums weren’t behaving like usual, and I had to dash off to the eclipse.
It would be really helpful for anyone with experience building servers with Express to look at the code!
Also, I would really appreciate it if someone else tackled writing a streaming file-system store. It should be straightforward; Node makes streaming to and from files easy.
