by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 10 2016 13:26
    martinwoodward unassigned #296
  • Nov 10 2016 13:26
    martinwoodward unassigned #295
  • Nov 10 2016 13:26
    martinwoodward unassigned #293
  • Nov 10 2016 13:26
    martinwoodward unassigned #294
  • Nov 10 2016 13:26
    martinwoodward unassigned #292
  • Nov 10 2016 13:26
    martinwoodward unassigned #291
  • Nov 10 2016 13:26
    martinwoodward unassigned #288
  • Nov 10 2016 13:26
    martinwoodward unassigned #286
  • Nov 10 2016 13:26
    martinwoodward unassigned #287
  • Nov 10 2016 13:26
    martinwoodward unassigned #285
  • Nov 10 2016 13:26
    martinwoodward unassigned #281
  • Nov 10 2016 13:26
    martinwoodward unassigned #280
  • Nov 10 2016 13:26
    martinwoodward unassigned #278
  • Nov 10 2016 13:26
    martinwoodward unassigned #273
  • Nov 10 2016 13:26
    martinwoodward unassigned #271
  • Nov 10 2016 13:26
    martinwoodward unassigned #276
  • Nov 10 2016 13:26
    martinwoodward unassigned #272
  • Nov 10 2016 13:26
    martinwoodward unassigned #275
  • Nov 10 2016 13:26
    martinwoodward unassigned #268
  • Nov 10 2016 13:26
    martinwoodward unassigned #265
Tommy Long
@smudge202
I had a "quick" try at checking Assembly.Load is available from CoreCLR. My attempt failed because I dug myself into a DNX hole, but I hold to the fact that I think it's possible (though the string overload is not, you need to use an AssemblyName if I remember correctly).
The question is which TFM you'd want to target. I was trying to run it against Profile 259, netcore45, dnxcore50, and dotnet1.1 (netstandard1.0). I'm pretty sure it will work in all cases, but I don't know which would be the best pick. dotnet1.1 is probably the best choice if it's possible.
Matt Warren
@mattwarren
sheesh, the numbering/naming scheme in DNS/CoreCLR is tricky!
"net standard 1.4 is actually dotnet5.5"!!
Tommy Long
@smudge202
Yes, there's an off-by-one difference between dotnet5.x vs netstandard. They decided to decrement to zero index (netstandard1.0 = dotnet5.1) when they decided upon netstandard being the way forward.
I'm going to take at the CoreCLR support. Just curious what it is we depend upon in Dia2Lib/System.Management/etc. how crucial it is, and how hard it will be to replace.
I assume it's all replaceable because MS have their own benchmarking library which I think either already supports dnx/coreclr or is not far off.
Andrey Akinshin
@AndreyAkinshin
@smudge202, we will move the diagnostic stuff to another assembly/package (like BenchmarkDotNet.Diagnostic). So, the core library will not have any strange dependencies.
Matt Warren
@mattwarren

Just curious what it is we depend upon in Dia2Lib/System.Management/etc. how crucial it is, and how hard it will be to replace.

This is so we can use ClrMD to get the assembly code of the benchmark, not sure if/when that library will be CoreCLR/DNX, it's probably low on the priority list!

Tommy Long
@smudge202
I'm not familiar with ClrMD. I figured looking into this I'd end up reading some blogs... :)
Matt Warren
@mattwarren

I assume it's all replaceable because MS have their own benchmarking library which I think either already supports dnx/coreclr or is not far off.

Are you talking about this one https://github.com/Microsoft/xunit-performance?

Tommy Long
@smudge202
Yes
Ah, I understand ClrMD a bit more now.
Matt Warren
@mattwarren
Take a look at the readme on https://github.com/Microsoft/clrmd
But basically it's a .NET wrapper for the SOS extension for WinDBG, so it lets you analyse memory dumps and other low-level stuff like that
Tommy Long
@smudge202
I need to follow the calls we make into ClrMD. I can't see any external references in the csproj? https://github.com/Microsoft/clrmd/blob/master/src/Microsoft.Diagnostics.Runtime/Microsoft.Diagnostics.Runtime.csproj
Does it P/Invoke into something win32?
(If you don't know, don't worry, am looking and asking questions at same time)
So I guess porting will be quite a problem
I can only assume that the xunit.performance variant doesn't dig so deeply into the diagnostic output?
Matt Warren
@mattwarren
At the moment I'm including a custom version of the code they use to host at https://github.com/Microsoft/dotnetsamples/tree/master/Microsoft.Diagnostics.Runtime/CLRMD/
They've since re-factored everything and I haven't updated it yet. But all the code is in https://github.com/PerfDotNet/BenchmarkDotNet/blob/master/BenchmarkDotNet/Toolchain/BenchmarkCodeExtractor.cs

I can only assume that the xunit.performance variant doesn't dig so deeply into the diagnostic output?

I don't think they have anything similar at the moment, I've been tracking that project for a while and they are mostly focussed on making perf tests that run as part of a C.I build and let you know if there have been any perf regressions

I also tried to tell them about BenchmarkDotNet and suggest that they use that instead!! But I've not heard a response yet ;-) See https://github.com/dotnet/roslyn/issues/670#issuecomment-158365614

Tommy Long
@smudge202
Yeh. Huge portions of this may never become available on CoreCLR
That's not the end of the world thinking about it...
hmm, not ideal either...
Matt Warren
@mattwarren

Yeah I wondered if that would be the case, although it'll be shame as it's a handily library for debugging memory dumps. Although you can always use SOS in WinDBG, it's just a bit more manual doing it that way

At least by loading it dynamically we can just refuse to load the "plug-in" dll if we're running on CoreCLR

Tommy Long
@smudge202
I'm thinking most people will have, as an example, an aspnet5 website that targets dnx451 and dnxcore50. Their test projects will commonly only target dnx451 or perhaps even a full net4*, which wouldn't be too much work to get working (just resolve the DNX issue).
However, I've been harassing MS and Brad Wilson to update the VS tooling to allow not only cross compilation/analysis, but also modifications to Test Explorer and runtime to allow cross-testing.
Brad can't do anything until the VS team sort their stuff out
But in my mind, if I write tests CoreCLR compatible, I want to be able to run tests against all the targets I can. There are often fundamental differences between runtimes, let alone taking into account any #ifdefs
I'd want the same for Benchmark, but seeing as tooling doesn't support it...
And that most will run under dnx4* or net*, we can probably just try to resolve the dnx451 compatibility?
Also, it's probably obvious, but WinDBG doesn't work on *nix, Rasberry Pi, yada, so there'll probably never work on making that stuff CoreCLR compat because the amount of maintenance/work it would introduce providing native assemblies across platforms.
Matt Warren
@mattwarren
re the Xunit-performance tool:
One advantage of BenchmarkDotNet, we don't require a modified unit test runner for us to work. For instance I just wrote an integration tests that shows how you can use us in a unit test, so that the test will fail if one benchmark doesn't run quicker than another. See https://github.com/PerfDotNet/BenchmarkDotNet/blob/master/BenchmarkDotNet.IntegrationTests/PerformanceUnitTest.cs#L24-L31

I'd want the same for Benchmark, but seeing as tooling doesn't support it...

To be fair, the feature that dumps the assembly is probably only useful when you are investigating a benchmark, in which case you'll probably be running it in a console. I don't see it being that useful when you want to run Benchmarks as part of (performance) unit tests

Tommy Long
@smudge202
True. Trying to keep up with the xunit runner is a nightmare. A friend of mine has been trying to maintain a set of custom xunit attributes and associated runners, discoverers, and various other pieces of black magic. But there are so few examples online of xunit extensibility that are valid given the current API, which of course is constantly rotating alongside DNX development.
Side note: Unit tests shouldn't take more than a few ms in my opinion. I'd never want benchmarking to run as part of my Ctrl+R, A ritual. :)
(Replace Ctrl+R, A with R# CI / NCrunch / etc)
Matt Warren
@mattwarren
Right back when @AndreyAkinshin and I first talked, we chatted about the 2 main scenarios for benchmarks:
1) when you are investigating something, so running it in a console, pasting results onto a GitHub issue, understanding why something is faster, etc
2) running perf tests as part of a C.I build, to show you any regressions, ensure that 1 piece of code always runs faster than another piece of code or faster than X milliseconds
I think we're currently very good for 1), but maybe need a bit of work for 2)
Tommy Long
@smudge202
Want me to describe the scenario I was facing when I stumbled upon you guys? :)
Matt Warren
@mattwarren

True. Trying to keep up with the xunit runner is a nightmare. A friend of mine has been trying to maintain a set of custom xunit attributes and associated runners, discoverers, and various other pieces of black magic. But there are so few examples online of xunit extensibility that are valid given the current API, which of course is constantly rotating alongside DNX development.

Exactly and you'd have to do that for every test-runner that people wanted to use, compared to just writing code like this https://github.com/PerfDotNet/BenchmarkDotNet/blob/master/BenchmarkDotNet.IntegrationTests/PerformanceUnitTest.cs#L24-L31

Tommy Long
@smudge202
Loosely fits with your second point I guess
Although the code in that snippet is really cool, it makes me wince because I hate the thought of running such code in a unit test. Probably just me though.
Matt Warren
@mattwarren

Side note: Unit tests shouldn't take more than a few ms in my opinion. I'd never want benchmarking to run as part of my Ctrl+R, A ritual. :)

Yeah I agree, I see performance tests as similar to integration tests, probably only run as part of a nightly C.I build, or when a developer explicitly wants to run them. I guess controlled via a [Trait] in Xunit

Tommy Long
@smudge202
Yeh, I'd use Traits / Categories.
Matt Warren
@mattwarren

Want me to describe the scenario I was facing when I stumbled upon you guys?

yes please

Tommy Long
@smudge202
I wish the MS attributes weren't sealed / more extensible. :'(
Matt Warren
@mattwarren

Although the code in that snippet is really cool, it makes me wince because I hate the thought of running such code in a unit test. Probably just me though.

Yeah, the only advantage of running them as part of a unit test it that it makes the C.I tooling, pass/fail metrics, failing the build. etc a bit easier. But it could just a easily be a separate .exe that is run via a script, like the BenchmarkDotNet.Samples exe