PerfDotNet/BenchmarkDotNet

Powerful .NET library for benchmarking

Andrey Akinshin

@AndreyAkinshin

By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.

Matt Warren

@mattwarren

It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.

Yeah good point, I guess we just add what we can to BenchmarkDotNet and see what happens. It's already getting pretty good traction in the CoreCLR/CoreFX GitHub issues.

By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.

That's cool, it'll stop things like this happening! https://twitter.com/ben_a_adams/status/678447298142199808

Jared Hester

@cloudRoutine

I've been using BenchmarkDotNet for a few weeks now and love it, PerfUtil used to be my goto until I found your project. Thanks for all your hard work! I've been trying to evangelize it to the F# community, and I think they're slowly warming to it. Maybe an FSharp.Charting plugin/helper module would make it more enticing....

Andrew Cherry

@kolektiv

@cloudRoutine - you've got it working with f#? i'm trying, but i'm finding it doesn't pick up an f# core. the assemblies and associated .config it generates doesn't have the assembly binding redirects to pick up FSharp.Core - have you found that? or a way round it?

Jared Hester

@cloudRoutine

@kolektiv I didn't run into that issue, got a repo I can inspect?

Jared Hester

@cloudRoutine

@kolektiv this trival example should run without issue

module BDNetFSharp.Program
open BenchmarkDotNet 
open System.Reflection

type ArrayVsList () =
    [<Params (10, 100)>] 
    member val public ColSize = 0 with get, set
    [<Benchmark>]
    member self.GetSquareArray () =
        let value = self.ColSize / 2 in [| for _ in 0 .. self.ColSize -> value |].[value]
    [<Benchmark>]
    member self.GetSquareList () =
        let value = self.ColSize / 2 in [for _ in 0 .. self.ColSize -> value ].[value]

let [<EntryPoint>] main args =
    let competitionSwitch =    
        Assembly.GetExecutingAssembly().GetTypes()
        |> Array.filter (fun t ->
            t.GetMethods (BindingFlags.Instance ||| BindingFlags.Public )
            |> Array.exists (fun m -> m.GetCustomAttributes (typeof<BenchmarkAttribute>, false) <> [||]))
        |> BenchmarkSwitcher
    competitionSwitch.Run args 
    0

Andrey Akinshin

@AndreyAkinshin

@cloudRoutine, I want to add special plugin type for plotting on the next week. It would be cool, if you can create PR with FSharp.Charting support.

Andrey Akinshin

@AndreyAkinshin

@mattwarren, check it out, I added BenchmarkEnvironmentAnalyser: PerfDotNet/BenchmarkDotNet@de01401

Andrew Cherry

@kolektiv

@cloudRoutine - see issue #67 :)

Andrey Akinshin

@AndreyAkinshin

Another project that uses BenchmarkDotNet: http://www.mapperator.net/Benchmark

Andrey Akinshin

@AndreyAkinshin

@mattwarren, I have implemented basic plot support (based on R script) in the refactoring branch.

Matt Warren

@mattwarren

Looks cool

Tommy Long

@smudge202

I've been assigned some issues? ^^

Andrey Akinshin

@AndreyAkinshin

@smudge202, I thought that you want to implement CoreCLR support (you even have a local branch with some commits: https://github.com/smudge202/BenchmarkDotNet/commits/features/33-dnx-compatibility) Am I wrong?

@mattwarren, I am ready to publish v0.8.2, there is ChangeLog: https://github.com/PerfDotNet/BenchmarkDotNet/wiki/ChangeLog It includes your recent Baseline feature. Do you want to commit something before publishing?

Tommy Long

@smudge202

I'll take a look when I get the chance, but it's not very high on a fairly long list I'm afraid. We're not using Benchmarks as we couldn't find a good solution for our use cases.

Andrey Akinshin

@AndreyAkinshin

Ok, then I do it myself.

Matt Warren

@mattwarren

@AndreyAkinshin I just pushed one extra change (PerfDotNet/BenchmarkDotNet@9e16e8c) to make the Delta Result Extender a bit more robust. If that could go into 0.8.2 as well that would be cool.

BTW I think that the ChangeLog is a really nice idea. It also shows that BenchmarkDotNet is becoming a more mature OSS project, good job!

Andrey Akinshin

@AndreyAkinshin

@mattwarren, Ok, no problem. I will test the last develop version tonight and publish it.

Andrey Akinshin

@AndreyAkinshin

@mattwarren, your last merge contains errors, I can't build our solution in the develop branch.

You have removed my BenchmarkStatResultExtenderTests and spoil BaselineDeltaResultExtenderTest, PerformanceUnitTest, see diff with parent 140deccc.

Andrey Akinshin

@AndreyAkinshin

I made a rollback. Please, review it.

Matt Warren

@mattwarren

Uggh, I'm sorry about that. It's weird I had a problem with the merge, but I thought that I double-checked it by downloading the latest code and building it from scratch

Did you manage to fix it, or do you need me to do something?

Andrey Akinshin

@AndreyAkinshin

I already fixed it.

Matt Warren

@mattwarren

thanks

Andrey Akinshin

@AndreyAkinshin

It would be cool, if you will check that everything is fine in the develop branch.

Matt Warren

@mattwarren

BTW can you wait to release till tomorrow night? I just noticed that "Baseline" doesn't work when you run a benchmark as new BenchmarkRunner().Run<BenchmarkProgram>();, it only works via the BenchmarkSwitcher

Yeah, I'll take a look at the develop branch as well

Andrey Akinshin

@AndreyAkinshin

can you wait to release till tomorrow night

Of course.

Matt Warren

@mattwarren

thanks

BTW this (me messing up the merge) makes me think that we need some sort of C.I build

I used AppVeyor before, can you access https://ci.appveyor.com/project/mattwarren/minibench-warren and see it?

Andrey Akinshin

@AndreyAkinshin

It is a very good idea.

Matt Warren

@mattwarren

Okay, I'll add an issue to track it

Andrey Akinshin

@AndreyAkinshin

can you access https://ci.appveyor.com/project/mattwarren/minibench-warren and see it?

Seems that is is a public page, everyone can see it.

Matt Warren

@mattwarren

okay, that might be because I'm on the free plan, I guess that doesn't matter for us does it? We could even put a badge on our GitHub page, showing the build status!!

Andrey Akinshin

@AndreyAkinshin

Yep.

Matt Warren

@mattwarren

can you wait to release till tomorrow night

I've just pushed the changes so that new BenchmarkRunner().Run<BenchmarkProgram>(); works when you have a Benchmark with [Benchmark(Baseline = true)]

so you can do the release now if you want

Andrey Akinshin

@AndreyAkinshin

Done:
https://github.com/PerfDotNet/BenchmarkDotNet
https://www.nuget.org/packages/BenchmarkDotNet/

Matt Warren

@mattwarren

cool

Andrey Akinshin

@AndreyAkinshin

@mattwarren, I really like ResultExtenders, but can I don't like the name. Do you have another idea about nice name for these classes? I want a one-word-name.

@mattwarren, also, what do you think about new following renaming: BenchmarkJitVersion -> BenchmarkJitter?

Matt Warren

@mattwarren

but can I don't like the name.

LOL yeah naming is always hard! I'll have a think about what single-word "ResultExtenders" could be change to

BenchmarkJitVersion -> BenchmarkJitter

Yeah makes sense to me. The only issue I see is that "Jitter" has another meaning (https://www.google.co.uk/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=jitter) which might confuse things

Where communities thrive

People

Repo info

Activity