Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 10 2016 13:26
    martinwoodward unassigned #296
  • Nov 10 2016 13:26
    martinwoodward unassigned #295
  • Nov 10 2016 13:26
    martinwoodward unassigned #293
  • Nov 10 2016 13:26
    martinwoodward unassigned #294
  • Nov 10 2016 13:26
    martinwoodward unassigned #292
  • Nov 10 2016 13:26
    martinwoodward unassigned #291
  • Nov 10 2016 13:26
    martinwoodward unassigned #288
  • Nov 10 2016 13:26
    martinwoodward unassigned #286
  • Nov 10 2016 13:26
    martinwoodward unassigned #287
  • Nov 10 2016 13:26
    martinwoodward unassigned #285
  • Nov 10 2016 13:26
    martinwoodward unassigned #281
  • Nov 10 2016 13:26
    martinwoodward unassigned #280
  • Nov 10 2016 13:26
    martinwoodward unassigned #278
  • Nov 10 2016 13:26
    martinwoodward unassigned #273
  • Nov 10 2016 13:26
    martinwoodward unassigned #271
  • Nov 10 2016 13:26
    martinwoodward unassigned #276
  • Nov 10 2016 13:26
    martinwoodward unassigned #272
  • Nov 10 2016 13:26
    martinwoodward unassigned #275
  • Nov 10 2016 13:26
    martinwoodward unassigned #268
  • Nov 10 2016 13:26
    martinwoodward unassigned #265
Matt Warren
@mattwarren

Also I have implemented new kind of plugins: analysers, see PerfDotNet/BenchmarkDotNet@527df65

I really like the new plugin system, especially the analysers, they're a really nice idea

Andrey Akinshin
@AndreyAkinshin
@mattwarren, thanks!
I created a wiki page: https://github.com/PerfDotNet/BenchmarkDotNet/wiki Feel free to make any changes.
Tommy Long
@smudge202
Good discussions in the repo today.
Nikita
@Metallium

what is the way to configure benchmark runner to make it less precise but run much faster?

it happens you often need to make raw A/B testing on some things

Andrey Akinshin
@AndreyAkinshin
@Metallium, right now you can reduce amount of processes (the first parameter of BenchmarkTask) , amount of warmup and target iterations (the two last parameters). E.g., processCount=1, warmupIterationCount=3, targetIterationCount=5. I am trying to implement smart logic that will be adjust these parameters based on you benchmark: if you really don't need super precision, benchmarking will perform quickly. Hopefully, new logic will be published in December.
Tommy Long
@smudge202
^ This would be great @AndreyAkinshin .
Btw, I took a look at the public API last night. I actually only found that one usage of Exec, everything else looked fine.
Andrey Akinshin
@AndreyAkinshin
This message was deleted
@smudge202, cool. Waiting for your PR.
Nikita
@Metallium
it would be great to flexible setup
already tried to lower iteration counts - benchmark still takes a lot)
Matt Warren
@mattwarren
You can also do this:
[BenchmarkTask(mode: BenchmarkMode.SingleRun, processCount: 1, warmupIterationCount: 1, targetIterationCount: 1)] i.e. "SingleRun"
Tommy Long
@smudge202
I haven't checked, but is it possible to inherit from BenchmarkTaskAttribute and assign default properties through the base ctor?
For example, allowing @Metallium to create a QuickBenchmarkTaskAttribute with settings as you've described.
Matt Warren
@mattwarren
good question, I'm not sure either?
Tommy Long
@smudge202
Pretty sure it isn't sealed. Will have to check the discoverer
Matt Warren
@mattwarren
well this compiles for me:
public class QuickBenchmarkTaskAttribute : BenchmarkTaskAttribute
    {
        public QuickBenchmarkTaskAttribute() 
            : base(mode: BenchmarkMode.SingleRun, processCount: 1, warmupIterationCount: 1, targetIterationCount: 1)
        { }
    }
Tommy Long
@smudge202
I don't think it will be discovered
Because the GetCustomAttributes call linked above states false for allow inheritance
Matt Warren
@mattwarren
and
    [QuickBenchmarkTask]
    public class PerformanceUnitTest
Tommy Long
@smudge202
https://msdn.microsoft.com/en-us/library/ms130871(v=vs.110).aspx <= we need to set inherit: true to discover derived attributes I believe
Matt Warren
@mattwarren
that's a good point about GetCustomAttributes, I haven't actually run the test yet¬
Tommy Long
@smudge202
I can't see a reason not to set it to true to be honest
I believe the code that's there will still work
Installing VS2015 U1 so can't check in VS, and too lazy to update VSCode... sorry. :)
Tommy Long
@smudge202
I'm going to create an associated bug. Either BenchmarkAttribute should be sealed or the GetCustomAttributes call should allow inheritance. The combination we have right now I consider to be a bug. Obviously, changing the call instead of sealing the attribute would be a better fix.
Matt Warren
@mattwarren
sounds good
Tommy Long
@smudge202
PerfDotNet/BenchmarkDotNet#57
Matt Warren
@mattwarren

wierdly enough

[QuickBenchmarkTask]
public class PerformanceUnitTest()

works for me, but I'll look into it a bit more when I get round to the issue (there's a bit of a backlog building up :-))

Tommy Long
@smudge202
Oh... was I wrong then? It does discover inherited attributes?
Matt Warren
@mattwarren
not really sure at the moment, maybe, maybe not!
Tommy Long
@smudge202
lol
Andrey Akinshin
@AndreyAkinshin
@smudge202, @mattwarren, do you have any comment about new plugin system and API? I want to make v0.8.0 on the weekend and publish it on NuGet. @smudge202, I am waiting four your PR with executables.
Andrey Akinshin
@AndreyAkinshin
Guys? Any feedback?
Matt Warren
@mattwarren
@AndreyAkinshin sorry for the delay, finally got time to look thru the plugin system, looks good to me
release it!
Andrey Akinshin
@AndreyAkinshin
@mattwarren, @smudge202, ok, v0.8.0 is in the NuGet. Now I want to keep the master branch in conformity with the latest NuGet package. Let's use the develop branch for developing.
Andrey Akinshin
@AndreyAkinshin
Guys, check it out: new benchmark framework https://petabridge.com/blog/introduction-to-nbench/ A funny thing: NBench uses BenchmarkDotNet for own infrastructure benchmarking: https://github.com/petabridge/NBench/blob/dev/benchmarks/NBench.Microbenchmarks/Program.cs
Matt Warren
@mattwarren
Cool, I saw NBench show up in my twitter feed, but didn't have time to look at it. That's really funny that NBench actually uses BenchmarkDotNet to test itself! (although I do wonder why he wrote a new library and didn't just use BenchmarkDotNet itself, maybe we weren't extensible enough? It wouldn't be that hard to add the NBench functionality to BenchmarkDotNet, now that we have the new plugin system)
Matt Warren
@mattwarren
Just found this comment from the author of NBench, it pretty much answers my previous question
Basicially this:
PerfDotNet only does throughput and micro-benchmarking, which is super helpful unto itself and it excels at doing that. But I needed to be able to also make assertions against Memory consumption and GC, so it didn't cover the spectrum of needs there. Plus I wanted unit test-style pass/fail semantics, which it also does not do.
Andrey Akinshin
@AndreyAkinshin
I published v0.8.1 on NuGet.
@mattwarren, please, use the develop branch for your commits. I want to keep the master branch and the NuGet package relevant.
Matt Warren
@mattwarren
yeah sorry about that, I just read the notice on the wiki page. I'll use develop in the future
what do you think about the NBench stuff, is it worth chatting to the author to see if we can persuade him to use BenchmarkDotNet, or do you think it's too late for that?
Andrey Akinshin
@AndreyAkinshin
Of course, we can discuss it. I think, if people really want to make performance unit tests, we can implement this feature in BenchmarkDotNet. However, we should put some warnings that they should do it carefully, use a separate machine for such tests, etc. For now, NBench has poor accuracy, our library has a lot of features that NBench doesn't have. It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.
By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.
Matt Warren
@mattwarren

It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.

Yeah good point, I guess we just add what we can to BenchmarkDotNet and see what happens. It's already getting pretty good traction in the CoreCLR/CoreFX GitHub issues.

By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.

That's cool, it'll stop things like this happening! https://twitter.com/ben_a_adams/status/678447298142199808