Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

scalameta/munit

People

Repo info

Activity

Gabriele Petronella

@gabro

ah, thanks for clarifying that @griggt 🙏 Looking forward to 3.0.1 then

Milan van der Meer

@milanvdm

Is there an out-of-the-box way to retry tests failing with a specific error?

Ólafur Páll Geirsson

@olafurpg

@milanvdm you can write a test transformer

if you want this behavior for all tests then it’s best to have an abstract super class that defines the test transformer, and all suites extend that class

https://scalameta.org/munit/docs/tests.html#customize-evaluation-of-tests-with-tags

Karthik Krishnamurthy

@kkrs

Hi, I'm trying to find how to run test suite like I'd do with scalacheck
using Prop.check for example

Gabriele Petronella

@gabro

@kkrs have you read the documentation of the scalacheck integration? Does it cover it?

Daniel Esik

@danicheg

Hi there. It would be awesome if someone could review this scalameta/munit#393. Thanks :pray:

Karthik Krishnamurthy

@kkrs

@gabro - i'm not looking to define property tests. Just want to know how I can run an munit test from main or an ammonite script for example. Prop.check allows me to do that.

Ólafur Páll Geirsson

@olafurpg

@kkrs this should probably be documented on the website. You can use JUnit main runners with MUnit

I don’t have a ready example but you should

be able to find some via google for how to run JUnit tests programmatically

Gerry Fletcher

@gerryfletch

Hi all. What would be the best way to extend the test function in FunFixtures to accept an additional implicit parameter, and pass it to the body of the test?

Essentially I have an F[Trace[F]], and I'd like to flatMap the Trace[F] into an implicit argument for the test body. I'm using cats-effect-munit.

Ólafur Páll Geirsson

@olafurpg

@gerryfletch the best way to achieve that is to implement a new method with a different name

Gerry Fletcher

@gerryfletch

I gave that a go but I was pretty tired; i'll try again!

Jens Halm

@jenshalm

Is it expected behaviour for munit that, when run with sbt and parallel execution on, it logs the tests of multiple suites intertwined and not, like ScalaTest for example, grouped by suite? Is there a way to change the configuration for this? It's something that could potentially stop me from migrating to munit.

Ólafur Páll Geirsson

@olafurpg

@jenshalm I think Test / testOptions += Tests.Argument(TestFrameworks.MUnit, "+l") configures MUnit to use the standard sbt loggers instead of println

it's disabled by default because sbt adds an [info] prefix to the output that messes up with how MUnit formats error messages

the +l name is cryptic, I can't remember exactly why it's not self-descriptive. The flag parsing logic started as a fork of sbt/junit-interface that uses single character flag names

Jens Halm

@jenshalm

Ok, thanks, I'll try that, but it sounds like it would just bring a different set of downsides. There is no way the munit implementation could change so that it does the proper grouping without sbt's logging? Meaning, is it something that would be difficult/impossible to implement, or just something that hasn't been done yet? If it's the latter I might try to look into it at some point. Good output in sbt is quite important for me as I cannot run the tests properly in IntelliJ either (their support for JVM/JS cross projects seems to be very poor - with any test framework)

Ólafur Páll Geirsson

@olafurpg

It's probably possible to implement a buffering logger outside of the sbt system that flushes the output after each test suite completes

You may need to implement the same logic twice, once in Scala and once in Java

not 100% sure if that's necessary here, but some internals are duplicate to support Scala.js and Native

Jens Halm

@jenshalm

Oh, that sounds fun. :-) Thanks for the pointers. I'll first check how quickly I can make sense of the implementation (I don't know much about JUnit or sbt's test framework, so it might take a while).

Ólafur Páll Geirsson

@olafurpg

the part that prints to stdout is here https://github.com/scalameta/munit/blob/0354373d463fc691a3560c2085d35d8783b5708b/junit-interface/src/main/java/munit/internal/junitinterface/RichLogger.java

we could add a flush() method that's called from EvenDispatcher https://github.com/scalameta/munit/blob/0354373d463fc691a3560c2085d35d8783b5708b/junit-interface/src/main/java/munit/internal/junitinterface/EventDispatcher.java

after every test suite has finished running

Jens Halm

@jenshalm

Isn't it more complicated though? Wouldn't the buffering need to happen with one buffer per suite in case of parallel execution?

Jens Halm

@jenshalm

I only briefly skimmed over the code in the two classes, I might take a closer look over the weekend. But I did not spot any easy opportunity to add this behaviour, as EventDispatcher only seems to have a way of detecting the first test of the suite and not the last.

Another little thing I noticed (although I might be missing something) is that RichLogger. popCurrentTestClassName() might be prone to race conditions in parallel execution.

Ólafur Páll Geirsson

@olafurpg

is that RichLogger. popCurrentTestClassName() might be prone to race conditions in parallel execution.

Good catch. That field can probably be removed because it's not really used

That code was originally forked from junit-interface, which had a different way to highlight relevant stack trace elements

I think you can buffer log messages in something like private final ConcurrentHashMap<Description, ConcurrentLinkedDeque<String>> buffers = new ConcurrentHashMap<>();

Ólafur Páll Geirsson

@olafurpg

And then override the testSuiteFinished hook to flush logs

System.out is already synchronized so everything should work in a concurrent setting as long as you use a call println() or write() once per test suite

Ólafur Páll Geirsson

@olafurpg

@jenshalm I took a stab at implementing this here https://github.com/scalameta/munit/compare/main...olafurpg:concurrent-logging?expand=1

feel free to commandeer that branch and open a PR

I'm open to enabling this by default

Jens Halm

@jenshalm

Ah great, I certainly would have needed much longer. I'll play with this on the weekend.

My main hurdle for quickly understanding the fairly simple code of RichLogger or EventDispatcher was merely the scoping, meaning how many instance of those exists, whether they are global, per suite, per test or some other scope. Which determines how they need to be synchronized (if at all). The use of synchronization in EventDispatcher suggests it is not "per suite", however it is only ever instantiated in JUnitTask.execute which kind of felt to me like it is scoped to a single suite.

Jens Halm

@jenshalm

@olafurpg did a quick check on my project and it seems to work great (apart from a minor issue of reverse order which is trivial to fix). I'll turn this into a PR over the weekend and also add some docs (I'd suggest to the existing section "Run tests in parallel" as it matters the most in that context).

Jens Halm

@jenshalm

btw. I think my confusion was justified. :-) Out of curiosity I added a lot of low-level logging to EventDispatcher and the result is: 1) I get one instance per suite, 2) the reportedSuites property never has more than one item in it (which is always the suite the dispatcher belongs to), 3) even with parallel execution each dispatcher is always invoked by the same thread. It's not super-important performance-wise, but from this observation it seems that some of the synchronization in EventDispatcher and RichLogger may not be necessary.

Ólafur Páll Geirsson

@olafurpg

Interesting

I only did the minimal changes to the original junit-interface implementation to tweak the error message reporting to my preferences

Looks like JUnitReporter is the Scala-equiavalent for EventDispatcher (they should probably be named the same)

I don't think we need to tweak the Scala implementation, it's only used for Scala.js and Native

Jens Halm

@jenshalm

Yes, I'll look into general improvements after I did the PR for buffered logging. I think EventDispatcher and RichLogger should be cleaned up. They are implemented for multi-suite, multi-thread use, but they are A) not used in that way in practice, B) would be buggy if used that way as there is a race condition.

Jens Halm

@jenshalm

@olafurpg I did the PR for buffered logging here: scalameta/munit#397 - I did not make this option the default yet, that's kind of your call, happy to add that change if you want. I'm also open to duplicate the work for JS and Native, maybe in a separate PR.

I also opened an issue for several concurrency issues I found in the junit-interface module (more than the ones I already mentioned here): scalameta/munit#399. It's easy to fix those in the way suggested in the ticket, but I did not start yet, as I wanted to check whether you have any objections. The work is also best done after buffered logging is merged.

Ross A. Baker

@rossabaker:matrix.org

[m]

I don't have a minimization yet, but http4s has lost its test output since upgrading to 0.7.28. I get it back if I turn off the buffered logging.

Ross A. Baker

@rossabaker:matrix.org

[m]

scalameta/munit#408

Chat via Matrix