Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 14:22
    MartinZec commented #872
  • 14:17
    MartinZec commented #872
  • 11:58
    NadeemBader edited #3738
  • 11:58
    NadeemBader opened #3738
  • 11:54
    CharliePoole assigned #793
  • 11:53
    CharliePoole commented #793
  • 11:40
    CharliePoole commented #872
  • 11:31
    MartinZec commented #872
  • 11:22
    CharliePoole commented #872
  • 10:26
    MartinZec commented #872
  • 07:51
    mikkelbu commented #542
  • 04:23
    manfred-brands commented #3736
  • 01:46
    CharliePoole commented #862
  • 01:13
    CharliePoole commented #860
  • 01:10
    CharliePoole commented #860
  • 01:06
    CharliePoole commented #861
  • 00:48
    CharliePoole commented #861
  • 00:25
    CharliePoole milestoned #874
  • 00:25
    CharliePoole labeled #874
  • 00:25
    CharliePoole labeled #874
Ryan French
@rfrenchy
hey, im using nunit 3.12, nunit3TestAdapter 3.17.0-beta.1 and dotnet core 3.1 to try and run a test with the [Explicit] attribute, but I am unable to run it. I am running the command dotnet test --filter TestCategory=Explicit. I've also tried variations of the filter argument to try and get it to run but with no luck. Is this a known issue? Or have i messed up the filter somehow.
Chris Maddock
@ChrisMaddock
@marius02 - sounds a little non-trivial, NUnit would have to intergrate with your source control system, right? There's nothing currently available, as far as I know.
@rfrenchy - I think the issue you're seeing is nunit/nunit3-vs-adapter#612 - there's an open PR to fix it
Ryan French
@rfrenchy
@ChrisMaddock aha, ty
Terje Sandstrom
@OsirisTerje
@rfrenchy The work is underway, and 3.17.0-beta 2 or 3 will have the fix in. Hope to get it out shortly. However, running using the category filter should have worked, because that is not using the Explicit feature @ChrisMaddock is talking about. Can you have a look at the NUnit.Where filter, and check if that works. Run it as dotnet test -- NUnit.Where Category=Explicit (It works on my repro) Btw, I can repro that the --filter doesn't work. It should... so that is a bug. Would you mind raising it at https://github.com/nunit/nunit3-vs-adapter/issues ?
Ace Olszowka
@aolszowka
Right now we use MSBuild's /m switch to build our solution in parallel (~3500 projects) afterwards we invoke NUnit Runner, however there is no reason why we shouldn't be able to have a task that runs immediately after the project is built to take advantage of the parallel processing. For some reason I thought that the NUnit-VS-Adapter did this, but everything I have seen looks like it is attempting to run after the build has been successful. I am going to try creating something myself, but did not want to reinvent the square wheel, am I just missing something?
Terje Sandstrom
@OsirisTerje
@aolszowka The adapter can enable parallelism of anything you give it, but it is started by the vstest executable. If your build chain waits for the compile step to be done before vstest is called, then you must split at that point. VSTest can enable parallell runs by itself, on assembly level. The adapter itself doesn't do anything wrt to that, but just passes everything down to the engine and framework.
Ace Olszowka
@aolszowka
@OsirisTerje yah that is what I was thinking, my familiarity with it is only from within the IDE itself, and we're familiar with the Parallel options, is there a target that can be called via MSBuild to run the tests that is publicly exposed?
Terje Sandstrom
@OsirisTerje
@aolszowka Which CI build system do you use? If you use Azure Devops, you have tasks there, but they will still run in a Compile - Test sequence. If I understand you correctly you need to split your system so that you can run each split in parallell, and each one consists of both compile and test. I'm pretty sure that is not something that can be done on build level, you need to set up your system and split/group according to the knowledge you have of your dependencies. I might not understand you completely though. With ~3500 projects you could generate a yml file that could handle this.
Ace Olszowka
@aolszowka

@OsirisTerje right now we use CruiseControl.NET.

It can be done within pure MSBuild here's an example of it being added:

  <Target Name="Test" AfterTargets="Build">
    <Exec Command="&quot;C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe&quot; &quot;$(TargetPath)&quot;" />
  </Target>

What will happen is when you build this in MSBuild after the "Build" target it will run the unit test, its a bummer because it loses an msbuild node while it builds but in our system this actually works to out, because we're building with some pretty beefy hardware (16 cores/ 32gb RAM per agent) and this would only occur on our unit test projects, the loss of a node is no big deal because we get throughput on nodes where they would otherwise be sitting doing nothing.

You can then have your CI gather up the TestLog.xml's at the end of the build process to parse for errors.

What would be nice though is to:

  1. Have this be shipped as a nuget package which has a .targets which performs this task, then we could quickly add it to any project which has a reference to NUnit
  2. Instead of using Exec which dumps its output to the console is have NUnit console either wrapped in an MSBuild task (I have done something similar already) or write against the NUnit framework directly to parse out an report errors/warnings in a standard MSBuild format, in this way you could even stop the build sooner rather than waiting for the whole thing to run though.

I am just trying to avoid reinventing the square wheel and was wondering if something like this already existed.

Terje Sandstrom
@OsirisTerje
@aolszowka MSBuild - many years since I used that for anything advanced... You have all the moving parts here, and you know your own situation, so I would just go and create your own nuget package to wrap this up. About stopping at first error, there is a StopOnError option which does that.
Chris Maddock
@ChrisMaddock
@aolszowka For MSBuild, I think https://github.com/loresoft/msbuildtasks is maybe what you're looking for? Be warned it's unmaintained.
Ace Olszowka
@aolszowka
@OsirisTerje Sounds fair thank you for the follow up, I was looking for more a confirmation that nothing like that already existed out of the box. Thank you!
@ChrisMaddock Yup I've actually contributed to that back in the day, thanks for the link, sad to hear that he is no longer maintaining it :(
Joseph Musser
@jnm2

@mikkelbu The other tests in the file have a level of indentation like this:

            var testCode = TestUtility.WrapClassInNamespaceAndAddUsing(@"
    class AnalyzeArgumentHasImplicitConversion
    {
        [TestCase(uint.MaxValue)]
        public void Test(long e) { }
    }");

If I add a using statement, how would you want that indented?

Rohithnm
@Rohithnm
Hi All, is there any way were we can re-execute OneTimeSetup on failure
Mikkel Nylander Bundgaard
@mikkelbu
@jnm2 I think it is fine to indent the using directive to the same level as the class. The reason for the indentation was so that the resulting code looked nicely formatted, but I'm not sure we have been that consistent with the indentation.
Chris Maddock
@ChrisMaddock
@Rohithnm - the short answer: "no"! :-) I'd recommend implementing any retry logic at that level yourself.
Rohithnm
@Rohithnm
@ChrisMaddock Thanks for the reply, it would be helpful if you could share any reference for this retry logic
Oleksandr Liakhevych
@Dreamescaper
@Rohithnm There is no built in logic, you'd need to implement it yourself. The simplest example would be some loop with try-catch inside, e.g.
for(int i = 0; i < 3; i++)
{
      try 
      {
             // Setup...
             return;
      }
      catch
      {
             // Handle error
      }
}
Rohithnm
@Rohithnm
@Dreamescaper Thanks for the response, i thought is there any specific class in Nunit which needs to be overridded with custom logic. as we do for the test attributes
almazmr
@almazmr

Hi, i am using VSTest to run tests via Azure DevOps. Tests run successfully, but using the option "rerun failed tests: true", an error appears during the restart phase.

vstest.console.exe "C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1\Tests.Web.dll"
/Settings:"C:\agent2.172.2\_work\_temp\3utv233tymm.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1"
/TestCaseFilter:"FullyQualifiedName=Tests.Web.Tests._5.CourierModuleTest.N1_SendingTest.Id_5_1_01_TransferToCourierModule(Chrome)|FullyQualifiedName=Tests.Web.Tests._3.IssuanceOfDocuments.InformationOnTheApplicationIdentificationOfRecipient.N2_RecipientIdentificationTest.Id_3_2_13_RegisterAddressByFiasTest(Chrome)"
NUnit Adapter 3.16.1.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

How can I fix this error and successfully restart the tests in DevOps?

Chris Maddock
@ChrisMaddock
@almazmr I'd suggest posting that as an issue on the nunit3-vs-adapter repo. The experts will be able to help there. :-)
@Rohithnm Yep - I agree with @Dreamescaper here - best to implement something in your own code to workaround this, as far as I can think.
Ace Olszowka
@aolszowka
@OsirisTerje @ChrisMaddock Back in July I mentioned a project to attempt to invoke NUnit3-Console shortly after the Test Project was built in MSBuild (after the Build target) as discussed above it did not seem like anything like that existed out of the box. Therefore I have created and pushed https://github.com/aolszowka/NUnit.MSBuild I wanted to solicit feedback from some of the thought leaders for NUnit to see if this is something that is already duplicated or something that is novel. I am using it internally with a lot of great results so far. This is MIT licensed and I am willing to respond to reasonable pull requests.
Mikkel Nylander Bundgaard
@mikkelbu
Can anyone point me in a direction of a project with a lot of NUnit3 tests and asserts, so we can test possible performance improvements to the NUnit analyzers. Currently, the "largest" I can think of is the NUnit project itself. See https://github.com/nunit/nunit.analyzers/pull/282#issuecomment-672288135 for some context.
Oleksandr Liakhevych
@Dreamescaper
Yep, I though about NUnit project itself as well :)
Julien
@LiohAu
Can anybody tell me if this binary : /Applications/Visual Studio.app/Contents/Resources/lib/monodevelop/AddIns/MonoDevelop.UnitTesting/NUnit3/NUnitRunner.exe is developped by the NUnit, Microsoft or MonoDevelop team ?
Chris Maddock
@ChrisMaddock
@LiohAu That's not an executable produced by the NUnit organization, as far as I'm aware.
@mikkelbu Try NodaTime perhaps? I think that's got a pretty sizable test library...and I wonder if @jskeet might even be interested in a contribution, if everything works well! :-)
@aolszowka Looks nice! I haven't personally used MSBuild properly in a good few years, but would have always relied on the MSBuildTasks tool before - it's good to have a replacement out there, if the former is no longer maintained!
Julien
@LiohAu
@OsirisTerje what happens behind the scenes when you select the .runsettings in the IDE ? where is the code that handles that ? in the NUnit VS adapter ? or in VS itself ?
Richard Adleta
@radleta

Running into a weird behavior trying to Debug Tests using VS 2019 16.7.2. I can get a breakpoint to stop but when I try to Continue or Step Into or Step Over it cancels the debugging session and outputs the below error. I can see a ThreadAbortException thrown as well. Any suggestions on how to fix or what I'm doing wrong?

Message: 
    Test cancelled by user
  Stack Trace: 
    RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
    RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
    RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
    Reflect.InvokeMethod(MethodInfo method, Object fixture, Object[] args)
    MethodWrapper.Invoke(Object fixture, Object[] args)
    TestMethodCommand.InvokeTestMethod(TestExecutionContext context)
    TestMethodCommand.RunTestMethod(TestExecutionContext context)
    TestMethodCommand.Execute(TestExecutionContext context)
    SimpleWorkItem.PerformWork()

Packages

    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit.Console" Version="3.11.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.17.0" />
Julien
@LiohAu
Is there an architecture diagram somewhere that would explain which NUnit omponent uses the other components ?
Julien
@LiohAu
At first I assumed that NUnit adapter was calling the NUnit console runner, but I don't see any reference to that .exe in the code, only references to vstest
so are NUnit Adapter and NUnit Console using vstest under the hood ?
Ace Olszowka
@aolszowka
@LiohAu I don't believe that there is a block diagram, NUnit Adapter is using the actual NUnit Engine under the hood, the NUnit Console also uses that same engine but has a CLI Front end to it.
Julien
@LiohAu
@aolszowka my concerns is with the runsettings file. I just followed the vstest code, and it seems that the runsettings is given to an MSBuild .target file, then it goes into the VsTest code, and it is then given to NUnit by the NUnit classes that respect the VSTest contrat.
the thing I am trying to understand is why runsettings is ignored when using VS Mac :(
Ace Olszowka
@aolszowka
VS Mac isn't the same as VS Proper I thought it was a reskinning of Android Studio or whatever that they purchased, it might be that they do not properly support the IRunSettings interface, are you setup to debug this?
Chris Maddock
@ChrisMaddock
@radleta Does your test have a timeout attirbute by any chance? This is fixed in master so the timeout doesn't apply when the debugger is attached, but it's not released yet
Julien
@LiohAu
@aolszowka behind VS mac, it is the monodevelop IDE and no i'm not setup to debug it ^^
But I assume everything is calling vstest.console.exe in the end. So if I can submit an issue or a PR on the NUnit MonoDevelop AddIn, to pass the NUnit file which is defined in the .csproj, that could be enough
Richard Adleta
@radleta
@ChrisMaddock Unfortunately, no it does not have a timeout attribute.
Terje Sandstrom
@OsirisTerje
@LiohAu The runsettings is sent down from whoever calls the adapter. I have sent a question to the MS product group about this, referring to the issue raised here nunit/nunit3-vs-adapter#778 . Please add whatever other comments you may have to that issue, MS will read it :-)
Julien
@LiohAu
@OsirisTerje It seems that people using .NET core projects in Windows based VS or in VS Mac/MonoDevelop are all depending on VsTest. But if you are not using a .NET core based project and that NUnit is in your project, the NUnit addin in VSMac/MonoDevelop, seems to fallback to an internal runner, and I did not find yet how to provide the settings...
Terje Sandstrom
@OsirisTerje
@LiohAu Just for others to see, MS has confirmed this doesn't yet work on VS Mac, see https://github.com/nunit/nunit3-vs-adapter/issues/778#issuecomment-679292768. They suggest a command line work around until the issue is resolved
Stephan
@zpqrtbnk
Quick Q before filing an issue... running tests with dotnet test ... and nunit3 adapter 3.16.1 does produce a complete Xml file, upgrading to adapter 3.17.0 produces an Xml file that only contains the <command-line> element - rings a bell / known issue?
Terje Sandstrom
@OsirisTerje
@zpqrtbnk No. Please raise an issue with repro :-)