These are chat archives for Snaipe/Criterion

13th
Oct 2016
Dominik
@kaidowei
Oct 13 2016 10:57
@Snaipe hi there :)
we just found a bug/strange behavior with 2.2.2
[----] Warning! The test "ar7config::scg_active3" crashed during its setup or teardown.
[----] Warning! The test "ar7config::scg_delete" crashed during its setup or teardown.
[----] Warning! The test "ar7config::vcc_no_scg1" crashed during its setup or teardown.
[----] Warning! The test "ar7config::vcc_no_scg2" crashed during its setup or teardown.
[====] Synthesis: Tested: 14 | Passing: 14 | Failing: 0 | Crashing: 0
hmm
The return Code of the Tests is 0
which is unfortunate, if we run it with jenkins
Franklin Mathieu
@Snaipe
Oct 13 2016 13:50
@kaidowei what are these tests doing during setup and/or teardown?
Also is this happening on v2.3.0-rc1?
Franklin Mathieu
@Snaipe
Oct 13 2016 14:01
Looking at your log in more detail, these tests are crashing during teardown. If a test crashes during setup it is immediately marked as failed.
Crashing during teardown does not impact the result of the test
Dominik
@kaidowei
Oct 13 2016 15:58
These tests tried to free something and due to a bug in our code, the free crashed.
Fortunately, we ran the tests manually, otherwise we would not have seen that.
so maybe crashing while teardown should also mark the tests as failed?
Franklin Mathieu
@Snaipe
Oct 13 2016 16:01
I'm not sure it would be a good design idea since if you enforce that everything has to go right in your teardown then why isn't it part of the test function itself? (Same reasoning as putting assertions in JUnit's @After is considered bad practice)
however I do agree that those warnings should mandate a nonzero exit status
perhaps make Criterion return a nonzero status when at least one warning or error is caught
and introduce something like --ignore-warnings for the original behaviour