nunit/nunit

The NUnit Framework is the oldest and most widely used .NET Test Framework

Terje Sandstrom

@OsirisTerje

Well, I think you can write it with constraint syntax too, splitting it up the same way. What is happening here is just nested asserts. I was justa bit lazy.... Assert.That(()=>
{
var x = wordCounter.CountWordRepetitions("");
Assert.That(x,Is.Not.Null);
Assert.That(x.Length,Is.EqualTo(0));
},Throws.Nothing);

The outer one will of course also scream when the inner one fails, but...

Eugene Krapivin

@EugeneKrapivin

yeah... I'd prefer to mix syntax in that case :D

thanks for the help though :)

Terje Sandstrom

@OsirisTerje

both ways will do that, btw... effect of the nesting.-... but I wouldn't bother about that. A failure is a failure. So if the messages gets out right, then... But imho, I think the non-constraint syntax is more clear in this case

try it, play around with it.. Adding further constraints to Nothing gets hard to get right, as you get the exception back... or null I think...

Eugene Krapivin

@EugeneKrapivin

I'm getting back the result of the ActualDelegate (which is in my case a Dictionary) however the constraints expect the delegate it self :D

Marius Morar

@marius02

Is there a way using NUnit to achieve the following scenario : for a [TestFixture] that has 10 scenarios to display(in console or in a logger doesn't matter now ) :

Running State : 1 0f 10 NameOfTheTest1....2 of 10 NameOfTheTest2....10 of 10 NameOfTheTest10  ?

Chris Maddock

@ChrisMaddock

@marius02 the --labels=Before flag is the closest there is, I believe. :-)

Marius Morar

@marius02

Is anyone familiar with this type of exception -> at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread():

10:15:43.298|0037|02|1|[Bad] Error during EndReactivate.|TimeoutException|The operation has timed out.
10:15:49.778|0011|02|1|[Bad] Error during EndReactivate.|TimeoutException|The operation has timed out.
10:16:02.272|0027|02|1|[Bad] Error during EndReactivate.|TimeoutException|The operation has timed out.
System.Transactions Critical: 0 : <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>vstest.executionengine.x86.exe</AppDomain><Exception><ExceptionType>System.Threading.ThreadAbortException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Thread was being aborted.</Message><StackTrace>   at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</StackTrace><ExceptionString>System.Threading.ThreadAbortException: Thread was being aborted.
   at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</ExceptionString></Exception></TraceRecord>

Chris Maddock

@ChrisMaddock

Hmm, no, new one of me. Any more information if the logs, if you run with --trace=Debug?

Marius Morar

@marius02

@ChrisMaddock unfortunately no , this is the error that is throw after each test is finish , and after that the execution engine is exited

The program '[7580] vstest.executionengine.x86.exe' has exited with code 0 (0x0).

Marius Morar

@marius02

@all is anybody using TestStack.BDDfy testing framework together with NUnit framework?

Eugene Krapivin

@EugeneKrapivin

Hey, quick question: in my code I'm going assembly scan for specific types

var handlers = Assembly
                .GetEntryAssembly()
                .GetReferencedAssemblies()
                .Select(Assembly.Load)
                .SelectMany(x => x.DefinedTypes)
                .Where(type => type.IsValidTransportHandler());

however when running tests in nunit, GetEntryAssembly is null and everything blows up.
I found in google and stack overflow questions dating 10 years ago... there is a solution, albeit a dirty one. However, it wouldn't work in dotnet core
any ideas?

Mikkel Nylander Bundgaard

@mikkelbu

Does Assembly.GetCallingAssembly work? Ps. How are you executing the test?

Eugene Krapivin

@EugeneKrapivin

Nunit runner from resharper on dev machines and nunit runner on the teamcity ci agents

GetCallingAssembly returns the nunit runner assembly...

Marius Morar

@marius02

Is there any attribute that I can use to a Setup method (like GlobalTimeSetUp and GlobalTimeTearDown) to be called just ones per multiple fixtures (at start up and at teardown in case you run multiple fixtures at the same time), because OneTimeSetUpAttribute and OneTimeTearDownAttribute is called for every Fixture/test class

Marius Morar

@marius02

Or I can achieve this by creating a custom attribute ?

Chris Maddock

@ChrisMaddock

@marius02 Take a look at SetUpFixture - which you can run per-namespace. That's the closest I can think of

ArsenShnurkov

@ArsenShnurkov

How to use nunit-console ? Various sources said that it is parallel, don't output failed tests immediately, and can lose xml log in case of crush. And this can't be resolved because it lose Events extension point after switching from NUnit v2 to NUnit v3. Sim I want some kind of guide with examples.

Marius Morar

@marius02

is there a way to add an LastUpdated custom attribute to a test (to hold a TimeSpan value maybe) to be updated each time when the test is modified ? if this can make sense at all

Marius Morar

@marius02

or maybe a crc comparison mechanism between different runs of the same tests ?

Marius Morar

@marius02

@ArsenShnurkov here you can find all the available command line options that you can use with nunit-console

here is an example of how I use it(one option)

"C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" D:\TestFolder\Test.dll --test="nsTestNamespace.nsSmokeTests.SmokeTestOne" --output=D:\TestFolder\TestOutput.txt --work=D:\TestFolder --result:TestResult.xml;format=nunit3

Ryan French

@rfrenchy

hey, im using nunit 3.12, nunit3TestAdapter 3.17.0-beta.1 and dotnet core 3.1 to try and run a test with the [Explicit] attribute, but I am unable to run it. I am running the command dotnet test --filter TestCategory=Explicit. I've also tried variations of the filter argument to try and get it to run but with no luck. Is this a known issue? Or have i messed up the filter somehow.

Chris Maddock

@ChrisMaddock

@marius02 - sounds a little non-trivial, NUnit would have to intergrate with your source control system, right? There's nothing currently available, as far as I know.

@rfrenchy - I think the issue you're seeing is nunit/nunit3-vs-adapter#612 - there's an open PR to fix it

Ryan French

@rfrenchy

@ChrisMaddock aha, ty

Terje Sandstrom

@OsirisTerje

@rfrenchy The work is underway, and 3.17.0-beta 2 or 3 will have the fix in. Hope to get it out shortly. However, running using the category filter should have worked, because that is not using the Explicit feature @ChrisMaddock is talking about. Can you have a look at the NUnit.Where filter, and check if that works. Run it as dotnet test -- NUnit.Where Category=Explicit (It works on my repro) Btw, I can repro that the --filter doesn't work. It should... so that is a bug. Would you mind raising it at https://github.com/nunit/nunit3-vs-adapter/issues ?

Ace Olszowka

@aolszowka

Right now we use MSBuild's /m switch to build our solution in parallel (~3500 projects) afterwards we invoke NUnit Runner, however there is no reason why we shouldn't be able to have a task that runs immediately after the project is built to take advantage of the parallel processing. For some reason I thought that the NUnit-VS-Adapter did this, but everything I have seen looks like it is attempting to run after the build has been successful. I am going to try creating something myself, but did not want to reinvent the square wheel, am I just missing something?

Terje Sandstrom

@OsirisTerje

@aolszowka The adapter can enable parallelism of anything you give it, but it is started by the vstest executable. If your build chain waits for the compile step to be done before vstest is called, then you must split at that point. VSTest can enable parallell runs by itself, on assembly level. The adapter itself doesn't do anything wrt to that, but just passes everything down to the engine and framework.

Ace Olszowka

@aolszowka

@OsirisTerje yah that is what I was thinking, my familiarity with it is only from within the IDE itself, and we're familiar with the Parallel options, is there a target that can be called via MSBuild to run the tests that is publicly exposed?

Terje Sandstrom

@OsirisTerje

@aolszowka Which CI build system do you use? If you use Azure Devops, you have tasks there, but they will still run in a Compile - Test sequence. If I understand you correctly you need to split your system so that you can run each split in parallell, and each one consists of both compile and test. I'm pretty sure that is not something that can be done on build level, you need to set up your system and split/group according to the knowledge you have of your dependencies. I might not understand you completely though. With ~3500 projects you could generate a yml file that could handle this.

Ace Olszowka

@aolszowka

@OsirisTerje right now we use CruiseControl.NET.

It can be done within pure MSBuild here's an example of it being added:

  <Target Name="Test" AfterTargets="Build">
    <Exec Command="&quot;C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe&quot; &quot;$(TargetPath)&quot;" />
  </Target>

What will happen is when you build this in MSBuild after the "Build" target it will run the unit test, its a bummer because it loses an msbuild node while it builds but in our system this actually works to out, because we're building with some pretty beefy hardware (16 cores/ 32gb RAM per agent) and this would only occur on our unit test projects, the loss of a node is no big deal because we get throughput on nodes where they would otherwise be sitting doing nothing.

You can then have your CI gather up the TestLog.xml's at the end of the build process to parse for errors.

What would be nice though is to:

Have this be shipped as a nuget package which has a .targets which performs this task, then we could quickly add it to any project which has a reference to NUnit
Instead of using Exec which dumps its output to the console is have NUnit console either wrapped in an MSBuild task (I have done something similar already) or write against the NUnit framework directly to parse out an report errors/warnings in a standard MSBuild format, in this way you could even stop the build sooner rather than waiting for the whole thing to run though.

I am just trying to avoid reinventing the square wheel and was wondering if something like this already existed.

Terje Sandstrom

@OsirisTerje

@aolszowka MSBuild - many years since I used that for anything advanced... You have all the moving parts here, and you know your own situation, so I would just go and create your own nuget package to wrap this up. About stopping at first error, there is a StopOnError option which does that.

Chris Maddock

@ChrisMaddock

@aolszowka For MSBuild, I think https://github.com/loresoft/msbuildtasks is maybe what you're looking for? Be warned it's unmaintained.

Ace Olszowka

@aolszowka

@OsirisTerje Sounds fair thank you for the follow up, I was looking for more a confirmation that nothing like that already existed out of the box. Thank you!

@ChrisMaddock Yup I've actually contributed to that back in the day, thanks for the link, sad to hear that he is no longer maintaining it :(

Joseph Musser

@jnm2

@mikkelbu The other tests in the file have a level of indentation like this:

            var testCode = TestUtility.WrapClassInNamespaceAndAddUsing(@"
    class AnalyzeArgumentHasImplicitConversion
    {
        [TestCase(uint.MaxValue)]
        public void Test(long e) { }
    }");

If I add a using statement, how would you want that indented?

Rohithnm

@Rohithnm

Hi All, is there any way were we can re-execute OneTimeSetup on failure

Mikkel Nylander Bundgaard

@mikkelbu

@jnm2 I think it is fine to indent the using directive to the same level as the class. The reason for the indentation was so that the resulting code looked nicely formatted, but I'm not sure we have been that consistent with the indentation.

Chris Maddock

@ChrisMaddock

@Rohithnm - the short answer: "no"! :-) I'd recommend implementing any retry logic at that level yourself.

Rohithnm

@Rohithnm

@ChrisMaddock Thanks for the reply, it would be helpful if you could share any reference for this retry logic

Oleksandr Liakhevych

@Dreamescaper

@Rohithnm There is no built in logic, you'd need to implement it yourself. The simplest example would be some loop with try-catch inside, e.g.

for(int i = 0; i < 3; i++)
{
      try 
      {
             // Setup...
             return;
      }
      catch
      {
             // Handle error
      }
}

Rohithnm

@Rohithnm

@Dreamescaper Thanks for the response, i thought is there any specific class in Nunit which needs to be overridded with custom logic. as we do for the test attributes

almazmr

@almazmr

Hi, i am using VSTest to run tests via Azure DevOps. Tests run successfully, but using the option "rerun failed tests: true", an error appears during the restart phase.

vstest.console.exe "C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1\Tests.Web.dll"
/Settings:"C:\agent2.172.2\_work\_temp\3utv233tymm.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1"
/TestCaseFilter:"FullyQualifiedName=Tests.Web.Tests._5.CourierModuleTest.N1_SendingTest.Id_5_1_01_TransferToCourierModule(Chrome)|FullyQualifiedName=Tests.Web.Tests._3.IssuanceOfDocuments.InformationOnTheApplicationIdentificationOfRecipient.N2_RecipientIdentificationTest.Id_3_2_13_RegisterAddressByFiasTest(Chrome)"
NUnit Adapter 3.16.1.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

How can I fix this error and successfully restart the tests in DevOps?

Chris Maddock

@ChrisMaddock

@almazmr I'd suggest posting that as an issue on the nunit3-vs-adapter repo. The experts will be able to help there. :-)

@Rohithnm Yep - I agree with @Dreamescaper here - best to implement something in your own code to workaround this, as far as I can think.

Ace Olszowka

@aolszowka

@OsirisTerje @ChrisMaddock Back in July I mentioned a project to attempt to invoke NUnit3-Console shortly after the Test Project was built in MSBuild (after the Build target) as discussed above it did not seem like anything like that existed out of the box. Therefore I have created and pushed https://github.com/aolszowka/NUnit.MSBuild I wanted to solicit feedback from some of the thought leaders for NUnit to see if this is something that is already duplicated or something that is novel. I am using it internally with a lot of great results so far. This is MIT licensed and I am willing to respond to reasonable pull requests.

Where communities thrive

People

Repo info

Activity