Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

CloudState/community

People

Repo info

Activity

Marcel Lanz

@marcellanz

https://observablehq.com/@marcellanz/cloudstate-go-tck-verification

The in-official "TCK Nightly Verification" this night for the first time failed.
😃 awesome

Adriano Santos

@sleipnir

I think a new tck has been released and some things have broken

Marcel Lanz

@marcellanz

cool... so things are happening

Adriano Santos

@sleipnir

This report should definitely be included directly in the main project. It is very good to have this visibility

Marcel Lanz

@marcellanz

There is a ticket to get structured output from the TCK. The nightly TCK run maintains a report available here:
https://gist.githubusercontent.com/marcellanz/a9cf0dffb01d1a83ba2bbf2ac52166fe/raw/cloudstate_go-support-tck-run.json

errors included.

I think LB has itself some quality checks with their CI running. Unfortunately I had not enough time to go further with this for the Community project.

Adriano Santos

@sleipnir

Yes, I imagine but this report is the best

Marcel Lanz

@marcellanz

It's implemented here: https://github.com/marcellanz/cloudstate/tree/nightly/tck_run/nightly

If anyone likes to pick up; I think a report via e-mail would be nice. It runs a travis job defined here:
https://github.com/marcellanz/cloudstate/blob/nightly/tck_run/.travis.yml

Adriano Santos

@sleipnir

cool

great work

Guy Youansi

@ralphlaude

@marcellanz good work!

Marcel Lanz

@marcellanz

thanks guys

… old stuff :)
It also seems to have a bug.

Adriano Santos

@sleipnir

I haven't been able to look at the logs yet, any suspicions?

Marcel Lanz

@marcellanz

This one seems to fail:
""verify get cart, add item, remove item, and failures"
https://github.com/cloudstateio/cloudstate/blob/master/tck/src/main/scala/io/cloudstate/tck/CloudStateTCK.scala#L456

those are all new tests by the work on the "new" TCK. Don't yet see why it fails.

Adriano Santos

@sleipnir

hummmm...

Thank's I'll evaluate later

Marcel Lanz

@marcellanz

Sure, actually I expect more issues, the original TCK was also kind of simple. The new tests are quite exhaustive. Great work by Peter.

Adriano Santos

@sleipnir

\0/

Thank you @pvlugter for breaking everything :D

That's good, now we have more robust tests

Marcel Lanz

@marcellanz

Also, I think he works on the CRDT TCK tests; they are even more comprehensive from what I saw with the functional tests being done.

Adriano Santos

@sleipnir

awesome

Marcel Lanz

@marcellanz

I would have really liked to work on the TCK for the CRDT model, but never got into Scala so far (I think I can read ir, but not write). Did all of them then in Go 😅

Adriano Santos

@sleipnir

Scala is difficult to read, but humanly possible, writing is another story. I don't think even skilled programmers in Scala know what they do for sure :D
Calm it was a joke

Marcel Lanz

@marcellanz

I just never learned it, can't say anything about it.

Peter Vlugter

@pvlugter

I'll take a look at the TCK failures. It looks like it's just against the native image proxy that it fails. This is tested in the cloudstate main repo integration tests. Interesting looking at the nightly build matrix: https://observablehq.com/@marcellanz/cloudstate-go-tck-verification as it has cloudstate-proxy-native-dev-mode:0.5.1 passing but cloudstate-proxy-native-dev-mode:latest failing. These are actually the same image (we don't have CI builds published of the native image versions): https://hub.docker.com/r/cloudstateio/cloudstate-proxy-native-dev-mode/tags — so it's weird that it would consistently pass on 0.5.1 but not on the same image with latest tag.

Adriano Santos

@sleipnir

Strange very strange

Thanks @pvlugter

Peter Vlugter

@pvlugter

Okay, I think I see what the issue is. It should actually fail for both the native and non-native proxy for 0.5.1. The TCK now tests failures in the shopping cart (that's the only change to that test) and the failure handling in the proxy was broken earlier so that it didn't return failures properly (but reset the connection). The TCK test uses the failure message from the client verified against what the proxy saw from the user function (we don't know what the exact failure message will be, varies for shopping cart implementations, but it checks these are the same).

@marcellanz, you may want to check the scripts, as I think it should be failing more in that TCK verification actually.

Adriano Santos

@sleipnir

Makes sense

Peter Vlugter

@pvlugter

So, to address this, we can either make the latest TCK handle the old proxy as well. Or we could always only verify TCK 0.5.1 and proxy 0.5.1 together, and TCK latest and proxy latest together (and I'll get a new native proxy published for latest).

Adriano Santos

@sleipnir

I think it makes sense to use the same version between proxy and tck

Peter Vlugter

@pvlugter

Yeah, agree

Adriano Santos

@sleipnir

Cool

Peter Vlugter

@pvlugter

I've manually published a new cloudstate-proxy-native-dev-mode:latest and added docker image publishing to the circleci build for the native-image proxies.

Peter Vlugter

@pvlugter

@marcellanz, and ignore what I said. The verification scripts do, of course, already match proxy and TCK version — that was just the extra combinations I ran locally, not the script. With the updated native-image proxy for latest, should all be good now.

Marcel Lanz

@marcellanz

cool, thanks for looking into it @pvlugter

Peter Vlugter

@pvlugter

So for the new TCK tests, it checks which test services are implemented, and marks tests as pending if they're not implemented. So it could be useful to include the pending tests in the TCK verification visualisation as well. Tried it out, with marking pending as orange:

Marcel Lanz

@marcellanz

nice 👍

Peter Vlugter

@pvlugter

https://observablehq.com/@pvlugter/cloudstate-tck-verification
pvlugter/cloudstate@a9ed85b

Adriano Santos

@sleipnir

Thanks for the work @pvlugter
Now let's work to make all the tests green

Viktor Klang (√)

@viktorklang

:+1: on that suggestion, Adriano! :D