System.Management/etc. how crucial it is, and how hard it will be to replace.
Just curious what it is we depend upon in Dia2Lib/System.Management/etc. how crucial it is, and how hard it will be to replace.
This is so we can use ClrMD to get the assembly code of the benchmark, not sure if/when that library will be CoreCLR/DNX, it's probably low on the priority list!
I assume it's all replaceable because MS have their own benchmarking library which I think either already supports dnx/coreclr or is not far off.
Are you talking about this one https://github.com/Microsoft/xunit-performance?
I can only assume that the xunit.performance variant doesn't dig so deeply into the diagnostic output?
I don't think they have anything similar at the moment, I've been tracking that project for a while and they are mostly focussed on making perf tests that run as part of a C.I build and let you know if there have been any perf regressions
I also tried to tell them about BenchmarkDotNet and suggest that they use that instead!! But I've not heard a response yet ;-) See https://github.com/dotnet/roslyn/issues/670#issuecomment-158365614
Yeah I wondered if that would be the case, although it'll be shame as it's a handily library for debugging memory dumps. Although you can always use SOS in WinDBG, it's just a bit more manual doing it that way
At least by loading it dynamically we can just refuse to load the "plug-in" dll if we're running on CoreCLR
net*, we can probably just try to resolve the
I'd want the same for Benchmark, but seeing as tooling doesn't support it...
To be fair, the feature that dumps the assembly is probably only useful when you are investigating a benchmark, in which case you'll probably be running it in a console. I don't see it being that useful when you want to run Benchmarks as part of (performance) unit tests
True. Trying to keep up with the xunit runner is a nightmare. A friend of mine has been trying to maintain a set of custom xunit attributes and associated runners, discoverers, and various other pieces of black magic. But there are so few examples online of xunit extensibility that are valid given the current API, which of course is constantly rotating alongside DNX development.
Exactly and you'd have to do that for every test-runner that people wanted to use, compared to just writing code like this https://github.com/PerfDotNet/BenchmarkDotNet/blob/master/BenchmarkDotNet.IntegrationTests/PerformanceUnitTest.cs#L24-L31
Side note: Unit tests shouldn't take more than a few ms in my opinion. I'd never want benchmarking to run as part of my Ctrl+R, A ritual. :)
Yeah I agree, I see performance tests as similar to integration tests, probably only run as part of a nightly C.I build, or when a developer explicitly wants to run them. I guess controlled via a [Trait] in Xunit
Although the code in that snippet is really cool, it makes me wince because I hate the thought of running such code in a unit test. Probably just me though.
Yeah, the only advantage of running them as part of a unit test it that it makes the C.I tooling, pass/fail metrics, failing the build. etc a bit easier. But it could just a easily be a separate .exe that is run via a script, like the BenchmarkDotNet.Samples exe