nunit/nunit

The NUnit Framework is the oldest and most widely used .NET Test Framework

Chris Maddock

@ChrisMaddock

Also, purely out of interest, what's your dependency that means you still need to use the nunit2 result format?

Marius Morar

@marius02

@ChrisMaddock thank you for your time, yes probably I will log the time from the assembly, and related to Nunit version I'm using right now 3.11 with nunit3 result format

CharliePoole

@CharliePoole

Here's a question that just hit me... For our nuget packages we rely on a relative path between the engine and installed extensions to find those extensions. The paths we provide assume that both the extensions and the console are installed in the same way - either using packages.config or PackageReference elements in the project.

Is it possible for a user to break us by having the console installed in one way and the extensions another way?

Chris Maddock

@ChrisMaddock

@marius02 No worries. :-)

@CharliePoole ...is that possible? Sounds horrible! :laughing:

CharliePoole

@CharliePoole

@ChrisMaddock I don't know if it is for sure, but I believe you could set up one project in a solution to use PackageReference and another to use packages.config. That would be a pretty dumb thing to do deliberately. This popped into my head while I was working on having the engine load extensions for running tests. Seems like with PackageReferences the engine in my output directory has to know where the cache is located. I can hard code that, but it will break for anyone who has the cache in a non-default location.

Ace Olszowka

@aolszowka

Saw the packages.config vs PackageReference this happens all the time to us, we ended up writing a CI task to ensure that we never use packages.config (PackageReference for us is FAR superior) but we have so much legacy code that gets ported up stuff slips though from time to time.

Chris Maddock

@ChrisMaddock

@CharliePoole Hmm...to my mind, if someone has a genuine reason to mix packages.config and PackageReference in the same solution, then they're in a niche enough situation they should be encouraged to write their own addins file. I think the default files should only support the majority cases. By the way, did I ever tell you about my nunit3-console install myExtension.dll idea? Another one I haven't found the time to write up yet. 😅

CharliePoole

@CharliePoole

I wasn't worried so much about someone with a genuine reason and smart enough to figure out how to do it. More wondering if somebody could stumble into it unintentionally. It would be a heck of a thing to debug.

Automating installation of the extensions isn't that hard. I do that in my package testing. But I don't know about having the console do it. It seems to me, if you need to run a command, why not just run nuget.exe or choco.exe? OTOH having a menu option in the GUI would be pretty cool!

CharliePoole

@CharliePoole

@/all I just released TestCentric GUI 1.4.0, with the ability to run both .NET Framework and .NET Core tests. Check it out on nuget.org or chocolatey.org.

Pieter Laenen

@pieterlaenen

Hi,

I am messing around with TheoryAttribute and found some peculiar behaviour:
When defining a Theory, there should always be a parameterless constructor, even when using TestFixtureAttribute with arguments.
Is the Theory attribute supposed to behave like that?

I made the following observations in the code below

[TestFixture] with [Theory] should always have a parameterless constructor, or will throw an exception (tested running dotnet test)
```
NUnit.Framework.Internal.InvalidTestFixtureException: {type} does not have a default constructor
```
[TestFixture] with [Theory] does not call constructors with parameters, even if they are present.

(slightly unrelated) Parameterless [TestFixture] is not instantiated when there are TestFixtureAttributes with parameters.

  [TestFixture]
  [TestFixture("attribute-argument")]
  [TestFixture("another-argument")]
  public class SingleArgumentTheory
  {
      [Datapoint]
      public string Argument;

      public SingleArgumentTheory() : this("constructor-argument")
      {
      }

      public SingleArgumentTheory(string argument)
      {
          Argument = argument;
      }

      [Theory]
      public void RunTestFixture(string value)
      {
          Console.Out.WriteLine($"Datapoint: {value}");
      }

  }

  [TestFixture("required-argument")]
  public class SingleArgumentTheoryWithoutParameterlessConstructor
  {

      [Datapoint]
      public string Argument;

      public SingleArgumentTheoryWithoutParameterlessConstructor(string argument)
      {
          Argument = argument;
      }

      [Theory]
      public void RunTestFixture(string value)
      {
          Console.Out.WriteLine($"Datapoint: {value}");
      }
  }

Mikkel Nylander Bundgaard

@mikkelbu

Is the Theory attribute supposed to behave like that?

I think it is a bug (the two first items - the last is more to be expected). At least I cannot find any place that indicates that it should behave in this way. You can create an issue at https://github.com/nunit/nunit/issues

Pieter Laenen

@pieterlaenen

Thank you, I will create the issue for the behaviour of [Type]

Using Rider and Visual Studio, I noticed that both have trouble visualising this exception, as it happens during the creation of the fixture. When using NUnitTestFixtureBuilder, it shows that the exception bubbles through beyond the BuildFrom method.

There exists Test#MakeInvalid, which could capture the failure reason, rather than just logging a 'stray 'exception during initialisation.
In a generalised case: should we not link all uncaught exceptions to some TestFixture, instead of logging them?

public class InitialiseFixture
    {
        [Test]
        public void Should_pass_argument_to_fixture()
        {
            // Arrange
            var argument = "test-argument";
            var arguments = new object[] {argument};
            var type = typeof(SingleArgumentTheory);

            // Act
            var fixture = new NUnitTestFixtureBuilder().BuildFrom(
                new TypeWrapper(type),
                new EmptyPreFilter(),
                new TestFixtureParameters(arguments)
            );

            // Assert
        }

        [TestFixture]
        private class SingleArgumentTheory
        {
            [Datapoint] public readonly string Argument;

            public SingleArgumentTheory(string argument)
            {
                Argument = argument;
            }

            [Theory]
            public void RunTestFixture(string value)
            {
                Console.Out.WriteLine($"Data point: {value}");
            }
        }

        private class EmptyPreFilter : IPreFilter
        {
            public bool IsMatch(Type type)
            {
                return true;
            }

            public bool IsMatch(Type type, MethodInfo method)
            {
                return true;
            }
        }
    }

Pieter Laenen

@pieterlaenen

In a generalised case: should we not link all uncaught exceptions to some TestFixture, instead of logging them?

Given that #3532 will be documented, rather than fixed, the question above becomes more relevant.

Concretely for the [TestFixture] with [Theory] without default constructor, it seems that the InvalidTestFixtureException could be added as a SkipReason on the fixture created by NUnitTestFixtureBuilder.

What do you think?

Mikkel Nylander Bundgaard

@mikkelbu

I think it sounds reasonable, but I've don't think I've worked in this area of the codebase, so I might overlook something

Marius Morar

@marius02

So I have the following nunit console option:

--test=nsSampleNamespaceOne.nsSampleNamespaceTwo.nsSampleNamespaceThird,nsSampleNamespaceOne.nsSampleNamespaceTwo.nsSampleNamespaceThird,nsSampleNamespaceOne.nsSampleNamespaceTwo.nsSampleNamespaceThird

In this case I will have a list of tests and after all the tests are finish we get the test results in the console output;
Is there a way to get the test results after each item in the test list? ex: test[0] started -> output results...test[1] started -> output results;

Mikkel Nylander Bundgaard

@mikkelbu

Not as far as I know. One alternative could be to call the console in a loop. But of course this will add some overhead, but give you the output for each iteration.

Chris Maddock

@ChrisMaddock

@marius02 add --labels=BeforeAndAfter, and see if that's what you're looking for. :slight_smile:

Ace Olszowka

@aolszowka

@OsirisTerje I am trying to get my feet wet with the NUnit3 Visual Studio Adapter, I am starting with this issue: nunit/nunit3-vs-adapter#658

I am deciphering your comment here:

"The adapter have two methods that the testhost can call, one "by sources" and one "by testcases". The first one here was originally meant to be called from 1) command line 2) Visual Studio when running All Tests. The second one was meant to be called only from Visual Studio when something less than All Tests were selected."

Reading the code and understanding more now how this works, it appears what you are saying is that previous to Visual Studio 2019 there was another way to enter into NUnit3TestExecutor.RunTests()? Can you outline where that was?

I think the ask of Microsoft is that we get more information in IRunContext to indicate how the test was invoked, is that a fair assessment of what you're asking for?

Ace Olszowka

@aolszowka

Ok diving down even more I think I might have found the answer to my question (this document is extremely useful https://github.com/microsoft/vstest-docs/blob/master/RFCs/0004-Adapter-Extensibility.md) it appears that you are saying in Visual Studio 2017 it called RunTests(IEnumerable<string> , IRunContext , IFrameworkHandle ) whereas now it only calls RunTests(IEnumerable<TestCase>, IRunContext, IFrameworkHandle)did I grok that correctly?

Eugene Krapivin

@EugeneKrapivin

Hey! got a quick question, failed to find an answer in the documentation: is there a way to continue assertion on the returned value when using Assert.That with a delegate? I'm trying to assert that there is no exception thrown and the returned value should match something (empty IEnumerable)

Terje Sandstrom

@OsirisTerje

You can use Assert.Multiple if I got you right

Eugene Krapivin

@EugeneKrapivin

            Assert.That(() => wordCounter.CountWordRepetitions(""), 
                Throws.Nothing.And.Not.Null.And.Empty);

the not null and not empty constraints fail due to

System.ArgumentException : Expected: Delegate But was: Dictionary`2 (Parameter 'actual')
   at NUnit.Framework.Internal.ConstraintUtils.RequireActual[T](Object actual, String paramName, Boolean allowNull)
   at NUnit.Framework.Constraints.ThrowsNothingConstraint.ApplyTo[TActual](TActual actual)
   at NUnit.Framework.Constraints.AndConstraint.ApplyTo[TActual](TActual actual)
   at NUnit.Framework.Constraints.Constraint.ApplyTo[TActual](ActualValueDelegate`1 del)
   at NUnit.Framework.Assert.That[TActual](ActualValueDelegate`1 del, IResolveConstraint expr, String message, Object[] args)
   at NUnit.Framework.Assert.That[TActual](ActualValueDelegate`1 del, IResolveConstraint expr)
   at interview_strings.tests.SolutionQualificationTests.Empty_Text_Should_Return_Empty(ICountWords wordCounter) in D:\code\Challanges\SplitCountChallange\tests\SolutionTests.cs:line 67

i guess the Not null and Empty constraints expect to get the Delegate, however they are getting the Delegate result

Terje Sandstrom

@OsirisTerje

What about something like: IEnumerable x;
Assert.Multiple( ()=>
{
Assert.DoesNotThrow(()=>x = wordCounter.CountWordRepetitions(""));
Assert.NotNull(x);
Assert.That(x.Length,Is.EqualTo(0));
});

Terje Sandstrom

@OsirisTerje

The Multiple is a bit off here though, but simplifying things can make it easier to read, perhaps trying like: Assert.DoesNotThrow(()=>
{
var x = wordCounter.CountWordRepetitions("");
Assert.NotNull(x);
Assert.That(x.Length,Is.EqualTo(0));
});

Eugene Krapivin

@EugeneKrapivin

I'm actually trying not to mix the constraint syntax and the classical syntax... but it does seem that the only way I've got to go here is to either mix or catch it manually and assert for it

looks like a missing feature really in the constraint syntax

Terje Sandstrom

@OsirisTerje

Well, I think you can write it with constraint syntax too, splitting it up the same way. What is happening here is just nested asserts. I was justa bit lazy.... Assert.That(()=>
{
var x = wordCounter.CountWordRepetitions("");
Assert.That(x,Is.Not.Null);
Assert.That(x.Length,Is.EqualTo(0));
},Throws.Nothing);

The outer one will of course also scream when the inner one fails, but...

Eugene Krapivin

@EugeneKrapivin

yeah... I'd prefer to mix syntax in that case :D

thanks for the help though :)

Terje Sandstrom

@OsirisTerje

both ways will do that, btw... effect of the nesting.-... but I wouldn't bother about that. A failure is a failure. So if the messages gets out right, then... But imho, I think the non-constraint syntax is more clear in this case

try it, play around with it.. Adding further constraints to Nothing gets hard to get right, as you get the exception back... or null I think...

Eugene Krapivin

@EugeneKrapivin

I'm getting back the result of the ActualDelegate (which is in my case a Dictionary) however the constraints expect the delegate it self :D

Marius Morar

@marius02

Is there a way using NUnit to achieve the following scenario : for a [TestFixture] that has 10 scenarios to display(in console or in a logger doesn't matter now ) :

Running State : 1 0f 10 NameOfTheTest1....2 of 10 NameOfTheTest2....10 of 10 NameOfTheTest10  ?

Chris Maddock

@ChrisMaddock

@marius02 the --labels=Before flag is the closest there is, I believe. :-)

Marius Morar

@marius02

Is anyone familiar with this type of exception -> at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread():

10:15:43.298|0037|02|1|[Bad] Error during EndReactivate.|TimeoutException|The operation has timed out.
10:15:49.778|0011|02|1|[Bad] Error during EndReactivate.|TimeoutException|The operation has timed out.
10:16:02.272|0027|02|1|[Bad] Error during EndReactivate.|TimeoutException|The operation has timed out.
System.Transactions Critical: 0 : <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>vstest.executionengine.x86.exe</AppDomain><Exception><ExceptionType>System.Threading.ThreadAbortException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Thread was being aborted.</Message><StackTrace>   at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</StackTrace><ExceptionString>System.Threading.ThreadAbortException: Thread was being aborted.
   at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</ExceptionString></Exception></TraceRecord>

Chris Maddock

@ChrisMaddock

Hmm, no, new one of me. Any more information if the logs, if you run with --trace=Debug?

Marius Morar

@marius02

@ChrisMaddock unfortunately no , this is the error that is throw after each test is finish , and after that the execution engine is exited

The program '[7580] vstest.executionengine.x86.exe' has exited with code 0 (0x0).

Marius Morar

@marius02

@all is anybody using TestStack.BDDfy testing framework together with NUnit framework?

Eugene Krapivin

@EugeneKrapivin

Hey, quick question: in my code I'm going assembly scan for specific types

var handlers = Assembly
                .GetEntryAssembly()
                .GetReferencedAssemblies()
                .Select(Assembly.Load)
                .SelectMany(x => x.DefinedTypes)
                .Where(type => type.IsValidTransportHandler());

however when running tests in nunit, GetEntryAssembly is null and everything blows up.
I found in google and stack overflow questions dating 10 years ago... there is a solution, albeit a dirty one. However, it wouldn't work in dotnet core
any ideas?

Mikkel Nylander Bundgaard

@mikkelbu

Does Assembly.GetCallingAssembly work? Ps. How are you executing the test?

Eugene Krapivin

@EugeneKrapivin

Nunit runner from resharper on dev machines and nunit runner on the teamcity ci agents

GetCallingAssembly returns the nunit runner assembly...

Marius Morar

@marius02

Is there any attribute that I can use to a Setup method (like GlobalTimeSetUp and GlobalTimeTearDown) to be called just ones per multiple fixtures (at start up and at teardown in case you run multiple fixtures at the same time), because OneTimeSetUpAttribute and OneTimeTearDownAttribute is called for every Fixture/test class

Marius Morar

@marius02

Or I can achieve this by creating a custom attribute ?

Chris Maddock

@ChrisMaddock

@marius02 Take a look at SetUpFixture - which you can run per-namespace. That's the closest I can think of

Where communities thrive

People

Repo info

Activity