Can anyone point me I the right direction here? thanks
Failed to get test method '' from assembly 'FirstAspNetCoyote, so I had better luck moving the Coyote stuff out to a new ClassLibrary which you can test separately, then add a reference to that from the FirstAspNetCoyote project.
I have a heavy background of using Microsoft Orleans so when I saw Coyote introductory I immediately start wondering how these libraries correlates to each other.
The test tooling is really awesome, and I've not seen anything like that on other actor frameworks (Orleans, Akka, Service Fabric Actors).
From what I saw in documentation materials, Coyote is hosted on a single process, i.e. executable. Is it capable of running on several nodes (different physical/virtual machines), so that Actors can call each other (like several Silos in Orleans combined into a Cluster, or Akka.Net)?
Maybe there's a document describing this already, I would really appreciate for the link to read.
Thank you!
2 replies
Hi @nanderto, if I understand correctly, you are trying to run a MSTest or xunit where the return type is a Microsoft.Coyote.Tasks.Task? This one https://github.com/nanderto/FirstAspNetCoyote/blob/master/SecondAspNetCoyoteTests/Pages/CounterViewModelTests.cs#L27, right? I don't think that is possible as MSTest or xunit does not understand our type (similar to how an async Main method cannot return a Coyote task) and require a regular C# task. We actually need to add this information to our documentation, but to run Coyote test programmatically (from inside a unit test like MSTest or xunit) and not using the CLI tool 'coyote', you need to do something like the following:
using Microsoft.Coyote.Specifications;
[TestMethod]
public async System.Threading.Tasks.Task IncrementCountTestAsnc()
{
// Create some test configuration, this is just a basic one using 100 test iteration.
// Each iteration executes the test method from scratch, exploring different interleavings.
Configuration configuration = Configuration.Create().WithTestingIterations(100);
// Create the Coyote runner programmatically.
// Assign the configuration and the test method to run.
TestingEngine engine = TestingEngine.Create(configuration, async () =>
{
var viewmodel = new SecondCoyoteLibrary.Pages.CounterViewModel(runtime);
viewmodel.CurrentCount = 10;
viewmodel.IncrementAmount = 3;
await viewmodel.IncrementCount();
Specification.Assert(viewmodel.CurrentCount == 13);
});
// Run the Coyote test.
engine.Run();
// Check for bugs.
Console.WriteLine($"Found #{engine.TestReport.NumOfFoundBugs} bugs.");
if (engine.TestReport.NumOfFoundBugs == 1)
{
Console.WriteLine($"Bug: {engine.TestReport.BugReports.First()}");
}
}Basically, you create a Coyote TestingEngine and pass to it a test configuration and a lambda (this lambda is using Coyote tasks). You then run it (which runs it like if you were running the Coyote tool from command line) and finally can check the report for bugs.
@lanekelly IIRC, the DGML feature currently only works with the Actors programming model, not with Tasks.
@lanekelly @akashlal thats correct, DGML tracing currently only works for actors (+ @lovettchris )
@pdeligia Thanks for taking a look. Yes I was returning a Coyote task instead of a System.Threading.Tasks task . Interestingly If I change to a System.Threading.Tasks then my code still compiles and runs in the website. This below is the method that is called and it returns a System.Threading.Tasks.Task
public async Task IncrementCount()
{
var request = new RequestEvent<int, int>(IncrementAmount);
AddActor = runtime.CreateActor(typeof(AddActor), request);
var response = await request.Completed.Task;
CurrentCount = CurrentCount + response;
runtime.SendEvent(AddActor, HaltEvent.Instance);
}This actually works in the website even though the requested.Completed.Task is a coyote task, but it does not work in the unit test.
The sample you provide also does not work as there is no runtime available to pass into the method. Besides which, I am not looking to replace the how the Coyote test engine works I just want to be able to run unit tests.
@nanderto sorry my code snippet was incomplete, this is how you pass the runtime:
TestingEngine engine = TestingEngine.Create(configuration, async runtime =>
{
...
});Btw, my code does not try to replace how the Coyote test engine works, it shows you how to run a Coyote test from inside another unit testing framework (MSTest, Xunit) using the Coyote systematic testing engine :)
You can think of the Coyote testing engine as its own "unit test runner for Coyote". The TestingEngine will create a special "systematic testing mode" Coyote runtime and pass it to your test via dependency injection. You are not supposed to create your own Coyote runtime during systematic testing (only in production).
E.g. see how tests are written here: https://github.com/microsoft/coyote-samples/blob/master/HelloWorldActors/Program.cs. Notice that a runtime is passed as input parameter (and not created by the user), when writing a test method using the Microsoft.Coyote.TestingServices.Test attribute (https://github.com/microsoft/coyote-samples/blob/master/HelloWorldActors/Program.cs#L38), but in the Main method you explicitly create a runtime (https://github.com/microsoft/coyote-samples/blob/master/HelloWorldActors/Program.cs#L16) since that is invoked in production.
However, using the Microsoft.Coyote.TestingServices.Test only works when running tests with our coyote test command line tool (see https://microsoft.github.io/coyote/learn/tools/testing). What I described above does the same thing but allows you to integrate with other unit testing frameworks like MSTest or Xunit.
Hope this clarifies your question?
@nanderto if you just want to run a regular unit test, without the systematic testing capability that coyote test gives you, then you can do this:
[TestMethod]
public async System.Threading.Tasks.Task IncrementCountTestAsnc()
{
ICoyoteRuntime runtime = new CoyoteRuntime();
var viewmodel = new SecondCoyoteLibrary.Pages.CounterViewModel(runtime);
viewmodel.CurrentCount = 10;
viewmodel.IncrementAmount = 3;
await viewmodel.IncrementCount();
Assert.AreEqual(13, viewmodel.CurrentCount);
}Notice how I changed the signature of the method to return a regular C# task. Thats all you need to run this. System tasks and Coyote tasks can work together in production :) But doing this approach will result in a flaky test as the concurrency is not controlled, whereas the approach that I described in my last post will result in Coyote controlling the concurrency and systematically exploring interleavings to find bugs in your unit test!
ConcurrentDictionary. I don't see any of its methods returning a native Task. For improved test coverage, make sure to insert a ExploreContextSwitch before calling any method on a ConcurrentDictionary. Let me look up the Channel type, haven't used it before ...
Channel. Ideally, you should mock it in the same way that we mocked TaskCompletionSource. But this implementation uses some internal APIs that make the mocking easy. We are going to consider making these APIs public so that your exercise will be simpler. For now, maybe you can have a simple mock.
1 reply
Btw, just to add to this, if you run the test from the command line tool (coyote test or coyote replay) you can add the option --break (-b) which will start the debugger before it runs the test (it basically instruments a System.Diagnostics.Debugger.Launch()). This typically opens a Visual Studio selector for the debug session, as @akashlal described. But I have only tried this approach of debugging using the VS IDE on Windows so far.
Another way, is to run the Coyote test programmatically. We plan to add documentation on this asap, but basically you can programmatically create a Coyote TestingEngine and run the test based on a test Configuration (the coyote command line tool uses this logic under the hood). That way, you can just debug the test with whatever IDE or process you are using until now (e.g. add breakpoints, right click the test and click debug). This is also useful if you want to run Coyote tests on a unit testing framework like xUnit or MSTest, etc. Until we add this info in our docs, you can see my answer on this closed GitHub issue that provides a code snippet how to do this (shows both test and replay): microsoft/coyote#23
ValueTask type yet, only the original Task type and other common types like TaskCompletionSource (please note that comparing to our actor/state-machine programming model, our task programming model is still in-preview, so we are keep adding new features and supported types, to make it easier to be consumed, and are prioritizing those based on user demand and feedback). Saying this, supporting ValueTask should not be too hard (as a lot of the logic to control and systematically test those is the same with Task), so we can try to add this asap.
Shutdown event, to which an actor will respond by serializing and storing its state, and then halting itself. There should not be any complications here, unless you have special requirements like shutting down actors without pausing client requests.
OnHaltAsync(Event e) and called runtime.Stop() which according to the docs "Terminates the runtime and notifies each active actor to halt execution." however OnHaltAsync was never called. Does that seem like a bug or should I move that to an issue?
OnHaltAsync method is called when an actor is about to halt. Note that halting of actors has to be done explicitly, by either sending the HaltEvent to it, or the actor itself doing a RaiseHaltEvent. Were you trying to halt an actor, but OnHaltAsync never got invoked? If so, do file an issue.
I checked the code and indeed Runtime Stop was not designed to notify all existing running actors via OnHaltAsync. it just does this :-)
public void Stop() => this.IsRunning = false;
But is an interesting idea, but we'd have to change the api to make Stop async, perhaps a new HaltAsync on ICoyoteRuntime would make sense, and this would be implemented by the ActorRuntime, and ignored by the task runtime. So how about this: microsoft/coyote#36