Repository: https://github.com/solid/test-suite - Code of Conduct: https://github.com/solid/specification#code-of-conduct
Ah yes, I have to publish a new release of banana-rdf for the new servers to work. I also need to get a relase for this milestone done very very soon.
@kjetilk ah yes, thanks! That does make me very happy :) I see, this line, right? https://github.com/solid/specification/blob/2b6652d31a15bb6921416573c2244324400fff45/protocol.html#L640
3 replies
When I run that script I see
=> [ 2/10] RUN git clone https://github.com/solid/node-solid-server 3.3s
=> [ 3/10] WORKDIR node-solid-server 0.1s
=> [ 4/10] RUN git checkout main 0.3s
=> [ 5/10] RUN npm install 75.2s
=> [ 6/10] RUN openssl req -new -x509 -days 365 -nodes -out ./server.cert -keyout ./server.key -subj "/C=RO/ST=Bucharest/L=Bucha 0.5se
=> [ 7/10] ADD config.json . 0.0s
=> [ 8/10] ADD config ./config 0.0s
=> [ 9/10] ADD data ./data 0.0s
=> [10/10] ADD .db ./.dbIt looks like its just checking out the current main branch of node-solid-server and running it. Is there any way I can run the test suite against the local instance?
@michielbdejong @RubenVerborgh Let's take it down a few notches =)
Michiel, Aaron's suggestions seem reasonable to me and will be easy going for everyone.
Let the information about implementations and their reports come from authoritative sources or at the very least be published given their consent. (FWIW, that's how it was for the LDN test suite.)
For instance for ESS, i'm having a hard time right now migrating https://github.com/solid/solid-crud-tests/blob/main/run-against-ess.sh#L10 from pod-compat.inrupt.com to pod.inrupt.com. We will no longer be able to run the WAC tests against ESS, but I was hoping we can still run the CRUD tests that way.
@jaxoncreed ah, running against a local install of NSS is easy:
- https://github.com/solid/webid-provider-tests/blob/main/example.sh#L4
- https://github.com/solid/solid-crud-tests/blob/main/run-against-nss.sh#L7 (adapt URL if you're running single-user mode locally)
- https://github.com/solid/web-access-control-tests/blob/main/run-against-nss.sh#L6
Re NSS CI running against main instead of against the current branch, you're right! I vaguely remember we had that as a todo item to fix that. I'm looking into it now.
For solid-nextcloud the CI was set up the proper way, using
ADD instead of git pull, see https://github.com/pdsinterop/solid-nextcloud/blob/main/Dockerfile#L17
And PSS CI also uses
ADD so that will also test the PR's branch correctly https://github.com/pdsinterop/php-solid-server/blob/main/Dockerfile#L20
@jaxoncreed let me know if I can help with https://github.com/solid/node-solid-server/pull/1611#issuecomment-901678729
Updated the CRUD tests 'run-against-ess' script from pod-compat.inrupt.com to pod.inrupt.com
Tests: 3 failed, 33 skipped, 33 passed, 69 total
All three failures have to do with concurrency. Does pod.inrupt.com support
If-None-Match: * request headers?
t support
If-Match and If-None-Match either yet, right?
are there plans to add that?
I’m trying to run the test suite locally to check it out, but I keep getting Error: Cannot find module ‘rdflib’. It’s linked in my node_modules. Is there something else I should do?
Try using https://GitHub.com/solid/solidos or just cd into the rdflib copy that causes the error and run
@gibsonf1 great!
npm run build there.@gibsonf1 great!
Is someone from Inrupt available to discuss the test results for pod.inrupt.com?
Regarding https://github.com/solid/solid-crud-tests/blob/main/test/surface/concurrency.test.ts we concluded that the spec does not require servers to pass these tests, so we'll still report them but clearly mark it as optional.
So 'WebID Provider', 'CRUD' (excluding Websockets-pub-sub and Concurrency), and 'WAC' will be required, and 'Concurrency', 'Websockets-Pub-Sub', and 'Monetization' will be optional. We also still want to add optional test suite sections about 'Websockets-Patch' and 'ACP', but these tests have yet to be written.
@ylebre @bourgeoa @edwardsph @csarven does that list sound correct, did I forget anything?
@kjetilk may want to comment as he has been going through the spec ensuring everything testable has an id: solid/specification#302
Just one comment, on the term CRUD, I have always been uncomfortable about using that in Solid context, for two reasons: They don't map all that well to HTTP methods, and secondly, but more importantly, they lack permissions subsets. Which is kind of a strength in some contexts, but when we need to be very careful to design a system that truly enables practical use of the principle of least privilege, which has big societal implications
anyway, that was probably pretty far out on a tangent :-)
@kjetilk good point, these tests used to be called the "storage" tests, to distinguish them as a testable server function from IDP function and from access control (WAC). We renamed them to CRUD because they don't test the storage backend implementation, but the layer just above that, in a sense. The test how resources are created, updated, represented and deleted. Hence the name :)
Or should I just create an issue on https://github.com/solid/community-server/issues ?
Basically, the 6 failures from https://github.com/solid/community-server/issues/902#issue-968511525 are stil happening in both the
main and the version/2.0.0 branch.
solid/specification#316 adds identifiers and descriptions to significant things like requirements, concepts.. in the Protocol. Same as in https://solid.github.io/web-access-control-spec/
We've been in the process of an architecture update for the last 2 days and noticed someone running the test-suite this morning, so that wouldn't run. The update is complete now on https://stage.graphmetrix.net