How to use nunit-console ? Various sources said that it is parallel, don't output failed tests immediately, and can lose xml log in case of crush. And this can't be resolved because it lose Events extension point after switching from NUnit v2 to NUnit v3. Sim I want some kind of guide with examples.
here is an example of how I use it(one option)
"C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" D:\TestFolder\Test.dll --test="nsTestNamespace.nsSmokeTests.SmokeTestOne" --output=D:\TestFolder\TestOutput.txt --work=D:\TestFolder --result:TestResult.xml;format=nunit3
hey, im using nunit 3.12, nunit3TestAdapter 3.17.0-beta.1 and dotnet core 3.1 to try and run a test with the
[Explicit] attribute, but I am unable to run it. I am running the command dotnet test --filter TestCategory=Explicit. I've also tried variations of the filter argument to try and get it to run but with no luck. Is this a known issue? Or have i messed up the filter somehow.
@rfrenchy - I think the issue you're seeing is nunit/nunit3-vs-adapter#612 - there's an open PR to fix it
@rfrenchy The work is underway, and 3.17.0-beta 2 or 3 will have the fix in. Hope to get it out shortly. However, running using the category filter should have worked, because that is not using the Explicit feature @ChrisMaddock is talking about. Can you have a look at the NUnit.Where filter, and check if that works. Run it as
dotnet test -- NUnit.Where Category=Explicit (It works on my repro) Btw, I can repro that the --filter doesn't work. It should... so that is a bug. Would you mind raising it at https://github.com/nunit/nunit3-vs-adapter/issues ?
Right now we use MSBuild's /m switch to build our solution in parallel (~3500 projects) afterwards we invoke NUnit Runner, however there is no reason why we shouldn't be able to have a task that runs immediately after the project is built to take advantage of the parallel processing. For some reason I thought that the NUnit-VS-Adapter did this, but everything I have seen looks like it is attempting to run after the build has been successful. I am going to try creating something myself, but did not want to reinvent the square wheel, am I just missing something?
@aolszowka The adapter can enable parallelism of anything you give it, but it is started by the vstest executable. If your build chain waits for the compile step to be done before vstest is called, then you must split at that point. VSTest can enable parallell runs by itself, on assembly level. The adapter itself doesn't do anything wrt to that, but just passes everything down to the engine and framework.
@aolszowka Which CI build system do you use? If you use Azure Devops, you have tasks there, but they will still run in a Compile - Test sequence. If I understand you correctly you need to split your system so that you can run each split in parallell, and each one consists of both compile and test. I'm pretty sure that is not something that can be done on build level, you need to set up your system and split/group according to the knowledge you have of your dependencies. I might not understand you completely though. With ~3500 projects you could generate a yml file that could handle this.
@OsirisTerje right now we use CruiseControl.NET.
It can be done within pure MSBuild here's an example of it being added:
<Target Name="Test" AfterTargets="Build">
<Exec Command=""C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "$(TargetPath)"" />
</Target>What will happen is when you build this in MSBuild after the "Build" target it will run the unit test, its a bummer because it loses an msbuild node while it builds but in our system this actually works to out, because we're building with some pretty beefy hardware (16 cores/ 32gb RAM per agent) and this would only occur on our unit test projects, the loss of a node is no big deal because we get throughput on nodes where they would otherwise be sitting doing nothing.
You can then have your CI gather up the TestLog.xml's at the end of the build process to parse for errors.
What would be nice though is to:
- Have this be shipped as a nuget package which has a .targets which performs this task, then we could quickly add it to any project which has a reference to NUnit
- Instead of using Exec which dumps its output to the console is have NUnit console either wrapped in an MSBuild task (I have done something similar already) or write against the NUnit framework directly to parse out an report errors/warnings in a standard MSBuild format, in this way you could even stop the build sooner rather than waiting for the whole thing to run though.
I am just trying to avoid reinventing the square wheel and was wondering if something like this already existed.
@aolszowka MSBuild - many years since I used that for anything advanced... You have all the moving parts here, and you know your own situation, so I would just go and create your own nuget package to wrap this up. About stopping at first error, there is a StopOnError option which does that.
@aolszowka For MSBuild, I think https://github.com/loresoft/msbuildtasks is maybe what you're looking for? Be warned it's unmaintained.
@ChrisMaddock Yup I've actually contributed to that back in the day, thanks for the link, sad to hear that he is no longer maintaining it :(
@mikkelbu The other tests in the file have a level of indentation like this:
var testCode = TestUtility.WrapClassInNamespaceAndAddUsing(@"
class AnalyzeArgumentHasImplicitConversion
{
[TestCase(uint.MaxValue)]
public void Test(long e) { }
}");If I add a using statement, how would you want that indented?
Hi, i am using VSTest to run tests via Azure DevOps. Tests run successfully, but using the option "rerun failed tests: true", an error appears during the restart phase.
vstest.console.exe "C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1\Tests.Web.dll"
/Settings:"C:\agent2.172.2\_work\_temp\3utv233tymm.runsettings"
/Logger:"trx"
/TestAdapterPath:"C:\agent2.172.2\_work\r1\a\UITest\drop\Tests.Web\bin\Release\netcoreapp3.1"
/TestCaseFilter:"FullyQualifiedName=Tests.Web.Tests._5.CourierModuleTest.N1_SendingTest.Id_5_1_01_TransferToCourierModule(Chrome)|FullyQualifiedName=Tests.Web.Tests._3.IssuanceOfDocuments.InformationOnTheApplicationIdentificationOfRecipient.N2_RecipientIdentificationTest.Id_3_2_13_RegisterAddressByFiasTest(Chrome)"
NUnit Adapter 3.16.1.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.How can I fix this error and successfully restart the tests in DevOps?
@Rohithnm Yep - I agree with @Dreamescaper here - best to implement something in your own code to workaround this, as far as I can think.
@OsirisTerje @ChrisMaddock Back in July I mentioned a project to attempt to invoke NUnit3-Console shortly after the Test Project was built in MSBuild (after the Build target) as discussed above it did not seem like anything like that existed out of the box. Therefore I have created and pushed https://github.com/aolszowka/NUnit.MSBuild I wanted to solicit feedback from some of the thought leaders for NUnit to see if this is something that is already duplicated or something that is novel. I am using it internally with a lot of great results so far. This is MIT licensed and I am willing to respond to reasonable pull requests.
Can anyone point me in a direction of a project with a lot of NUnit3 tests and asserts, so we can test possible performance improvements to the NUnit analyzers. Currently, the "largest" I can think of is the NUnit project itself. See https://github.com/nunit/nunit.analyzers/pull/282#issuecomment-672288135 for some context.
@mikkelbu Try NodaTime perhaps? I think that's got a pretty sizable test library...and I wonder if @jskeet might even be interested in a contribution, if everything works well! :-)
@aolszowka Looks nice! I haven't personally used MSBuild properly in a good few years, but would have always relied on the MSBuildTasks tool before - it's good to have a replacement out there, if the former is no longer maintained!
@mikkelbu Look at https://github.com/OsirisTerje/NUnit.VisualStudioAdapter.TestPrograms , autogenerated 50000 tests, or this https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue497
Running into a weird behavior trying to Debug Tests using VS 2019 16.7.2. I can get a breakpoint to stop but when I try to Continue or Step Into or Step Over it cancels the debugging session and outputs the below error. I can see a ThreadAbortException thrown as well. Any suggestions on how to fix or what I'm doing wrong?
Message:
Test cancelled by user
Stack Trace:
RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Reflect.InvokeMethod(MethodInfo method, Object fixture, Object[] args)
MethodWrapper.Invoke(Object fixture, Object[] args)
TestMethodCommand.InvokeTestMethod(TestExecutionContext context)
TestMethodCommand.RunTestMethod(TestExecutionContext context)
TestMethodCommand.Execute(TestExecutionContext context)
SimpleWorkItem.PerformWork()Packages
<PackageReference Include="NUnit" Version="3.12.0" />
<PackageReference Include="NUnit.Console" Version="3.11.1" />
<PackageReference Include="NUnit3TestAdapter" Version="3.17.0" />
so are NUnit Adapter and NUnit Console using vstest under the hood ?
the thing I am trying to understand is why runsettings is ignored when using VS Mac :(
But I assume everything is calling vstest.console.exe in the end. So if I can submit an issue or a PR on the NUnit MonoDevelop AddIn, to pass the NUnit file which is defined in the .csproj, that could be enough