I'm going to create an associated bug. Either
BenchmarkAttribute should be sealed or the GetCustomAttributes call should allow inheritance. The combination we have right now I consider to be a bug. Obviously, changing the call instead of sealing the attribute would be a better fix.
Guys, check it out: new benchmark framework https://petabridge.com/blog/introduction-to-nbench/ A funny thing: NBench uses BenchmarkDotNet for own infrastructure benchmarking: https://github.com/petabridge/NBench/blob/dev/benchmarks/NBench.Microbenchmarks/Program.cs
Cool, I saw NBench show up in my twitter feed, but didn't have time to look at it. That's really funny that NBench actually uses BenchmarkDotNet to test itself! (although I do wonder why he wrote a new library and didn't just use BenchmarkDotNet itself, maybe we weren't extensible enough? It wouldn't be that hard to add the NBench functionality to BenchmarkDotNet, now that we have the new plugin system)
Basicially this:
PerfDotNet only does throughput and micro-benchmarking, which is super helpful unto itself and it excels at doing that. But I needed to be able to also make assertions against Memory consumption and GC, so it didn't cover the spectrum of needs there. Plus I wanted unit test-style pass/fail semantics, which it also does not do.
@mattwarren, please, use the
develop branch for your commits. I want to keep the master branch and the NuGet package relevant.
what do you think about the NBench stuff, is it worth chatting to the author to see if we can persuade him to use BenchmarkDotNet, or do you think it's too late for that?
Of course, we can discuss it. I think, if people really want to make performance unit tests, we can implement this feature in BenchmarkDotNet. However, we should put some warnings that they should do it carefully, use a separate machine for such tests, etc. For now, NBench has poor accuracy, our library has a lot of features that NBench doesn't have. It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.
By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.
It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.
Yeah good point, I guess we just add what we can to BenchmarkDotNet and see what happens. It's already getting pretty good traction in the CoreCLR/CoreFX GitHub issues.
By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.
That's cool, it'll stop things like this happening! https://twitter.com/ben_a_adams/status/678447298142199808
I've been using BenchmarkDotNet for a few weeks now and love it, PerfUtil used to be my goto until I found your project. Thanks for all your hard work! I've been trying to evangelize it to the F# community, and I think they're slowly warming to it. Maybe an FSharp.Charting plugin/helper module would make it more enticing....
@kolektiv this trival example should run without issue
module BDNetFSharp.Program
open BenchmarkDotNet
open System.Reflection
type ArrayVsList () =
[<Params (10, 100)>]
member val public ColSize = 0 with get, set
[<Benchmark>]
member self.GetSquareArray () =
let value = self.ColSize / 2 in [| for _ in 0 .. self.ColSize -> value |].[value]
[<Benchmark>]
member self.GetSquareList () =
let value = self.ColSize / 2 in [for _ in 0 .. self.ColSize -> value ].[value]
let [<EntryPoint>] main args =
let competitionSwitch =
Assembly.GetExecutingAssembly().GetTypes()
|> Array.filter (fun t ->
t.GetMethods (BindingFlags.Instance ||| BindingFlags.Public )
|> Array.exists (fun m -> m.GetCustomAttributes (typeof<BenchmarkAttribute>, false) <> [||]))
|> BenchmarkSwitcher
competitionSwitch.Run args
0
@mattwarren, check it out, I added BenchmarkEnvironmentAnalyser: PerfDotNet/BenchmarkDotNet@de01401
@smudge202, I thought that you want to implement CoreCLR support (you even have a local branch with some commits: https://github.com/smudge202/BenchmarkDotNet/commits/features/33-dnx-compatibility) Am I wrong?
@mattwarren, I am ready to publish v0.8.2, there is ChangeLog: https://github.com/PerfDotNet/BenchmarkDotNet/wiki/ChangeLog It includes your recent
Baseline feature. Do you want to commit something before publishing?
BTW I think that the ChangeLog is a really nice idea. It also shows that BenchmarkDotNet is becoming a more mature OSS project, good job!
You have removed my
BenchmarkStatResultExtenderTests and spoil BaselineDeltaResultExtenderTest, PerformanceUnitTest, see diff with parent 140deccc.
Did you manage to fix it, or do you need me to do something?