Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 10 2016 13:26
    martinwoodward unassigned #296
  • Nov 10 2016 13:26
    martinwoodward unassigned #295
  • Nov 10 2016 13:26
    martinwoodward unassigned #293
  • Nov 10 2016 13:26
    martinwoodward unassigned #294
  • Nov 10 2016 13:26
    martinwoodward unassigned #292
  • Nov 10 2016 13:26
    martinwoodward unassigned #291
  • Nov 10 2016 13:26
    martinwoodward unassigned #288
  • Nov 10 2016 13:26
    martinwoodward unassigned #286
  • Nov 10 2016 13:26
    martinwoodward unassigned #287
  • Nov 10 2016 13:26
    martinwoodward unassigned #285
  • Nov 10 2016 13:26
    martinwoodward unassigned #281
  • Nov 10 2016 13:26
    martinwoodward unassigned #280
  • Nov 10 2016 13:26
    martinwoodward unassigned #278
  • Nov 10 2016 13:26
    martinwoodward unassigned #273
  • Nov 10 2016 13:26
    martinwoodward unassigned #271
  • Nov 10 2016 13:26
    martinwoodward unassigned #276
  • Nov 10 2016 13:26
    martinwoodward unassigned #272
  • Nov 10 2016 13:26
    martinwoodward unassigned #275
  • Nov 10 2016 13:26
    martinwoodward unassigned #268
  • Nov 10 2016 13:26
    martinwoodward unassigned #265
Matt Warren
@mattwarren
You can also do this:
[BenchmarkTask(mode: BenchmarkMode.SingleRun, processCount: 1, warmupIterationCount: 1, targetIterationCount: 1)] i.e. "SingleRun"
Tommy Long
@smudge202
I haven't checked, but is it possible to inherit from BenchmarkTaskAttribute and assign default properties through the base ctor?
For example, allowing @Metallium to create a QuickBenchmarkTaskAttribute with settings as you've described.
Matt Warren
@mattwarren
good question, I'm not sure either?
Tommy Long
@smudge202
Pretty sure it isn't sealed. Will have to check the discoverer
Matt Warren
@mattwarren
well this compiles for me:
public class QuickBenchmarkTaskAttribute : BenchmarkTaskAttribute
    {
        public QuickBenchmarkTaskAttribute() 
            : base(mode: BenchmarkMode.SingleRun, processCount: 1, warmupIterationCount: 1, targetIterationCount: 1)
        { }
    }
Tommy Long
@smudge202
I don't think it will be discovered
Because the GetCustomAttributes call linked above states false for allow inheritance
Matt Warren
@mattwarren
and
    [QuickBenchmarkTask]
    public class PerformanceUnitTest
Tommy Long
@smudge202
https://msdn.microsoft.com/en-us/library/ms130871(v=vs.110).aspx <= we need to set inherit: true to discover derived attributes I believe
Matt Warren
@mattwarren
that's a good point about GetCustomAttributes, I haven't actually run the test yet¬
Tommy Long
@smudge202
I can't see a reason not to set it to true to be honest
I believe the code that's there will still work
Installing VS2015 U1 so can't check in VS, and too lazy to update VSCode... sorry. :)
Tommy Long
@smudge202
I'm going to create an associated bug. Either BenchmarkAttribute should be sealed or the GetCustomAttributes call should allow inheritance. The combination we have right now I consider to be a bug. Obviously, changing the call instead of sealing the attribute would be a better fix.
Matt Warren
@mattwarren
sounds good
Tommy Long
@smudge202
PerfDotNet/BenchmarkDotNet#57
Matt Warren
@mattwarren

wierdly enough

[QuickBenchmarkTask]
public class PerformanceUnitTest()

works for me, but I'll look into it a bit more when I get round to the issue (there's a bit of a backlog building up :-))

Tommy Long
@smudge202
Oh... was I wrong then? It does discover inherited attributes?
Matt Warren
@mattwarren
not really sure at the moment, maybe, maybe not!
Tommy Long
@smudge202
lol
Andrey Akinshin
@AndreyAkinshin
@smudge202, @mattwarren, do you have any comment about new plugin system and API? I want to make v0.8.0 on the weekend and publish it on NuGet. @smudge202, I am waiting four your PR with executables.
Andrey Akinshin
@AndreyAkinshin
Guys? Any feedback?
Matt Warren
@mattwarren
@AndreyAkinshin sorry for the delay, finally got time to look thru the plugin system, looks good to me
release it!
Andrey Akinshin
@AndreyAkinshin
@mattwarren, @smudge202, ok, v0.8.0 is in the NuGet. Now I want to keep the master branch in conformity with the latest NuGet package. Let's use the develop branch for developing.
Andrey Akinshin
@AndreyAkinshin
Guys, check it out: new benchmark framework https://petabridge.com/blog/introduction-to-nbench/ A funny thing: NBench uses BenchmarkDotNet for own infrastructure benchmarking: https://github.com/petabridge/NBench/blob/dev/benchmarks/NBench.Microbenchmarks/Program.cs
Matt Warren
@mattwarren
Cool, I saw NBench show up in my twitter feed, but didn't have time to look at it. That's really funny that NBench actually uses BenchmarkDotNet to test itself! (although I do wonder why he wrote a new library and didn't just use BenchmarkDotNet itself, maybe we weren't extensible enough? It wouldn't be that hard to add the NBench functionality to BenchmarkDotNet, now that we have the new plugin system)
Matt Warren
@mattwarren
Just found this comment from the author of NBench, it pretty much answers my previous question
Basicially this:
PerfDotNet only does throughput and micro-benchmarking, which is super helpful unto itself and it excels at doing that. But I needed to be able to also make assertions against Memory consumption and GC, so it didn't cover the spectrum of needs there. Plus I wanted unit test-style pass/fail semantics, which it also does not do.
Andrey Akinshin
@AndreyAkinshin
I published v0.8.1 on NuGet.
@mattwarren, please, use the develop branch for your commits. I want to keep the master branch and the NuGet package relevant.
Matt Warren
@mattwarren
yeah sorry about that, I just read the notice on the wiki page. I'll use develop in the future
what do you think about the NBench stuff, is it worth chatting to the author to see if we can persuade him to use BenchmarkDotNet, or do you think it's too late for that?
Andrey Akinshin
@AndreyAkinshin
Of course, we can discuss it. I think, if people really want to make performance unit tests, we can implement this feature in BenchmarkDotNet. However, we should put some warnings that they should do it carefully, use a separate machine for such tests, etc. For now, NBench has poor accuracy, our library has a lot of features that NBench doesn't have. It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.
By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.
Matt Warren
@mattwarren

It is cheaper to add unit testing for BenchmarkDotNet instead of implementing another full-featured benchmarking library.

Yeah good point, I guess we just add what we can to BenchmarkDotNet and see what happens. It's already getting pretty good traction in the CoreCLR/CoreFX GitHub issues.

By the way, right now I am implementing new algorithms that will choose wise amount of iteration. It will allow to run benchmarks really quickly and still have good accuracy. I'm almost finished, but such things takes a lot of time for testing.

That's cool, it'll stop things like this happening! https://twitter.com/ben_a_adams/status/678447298142199808

Jared Hester
@cloudRoutine
I've been using BenchmarkDotNet for a few weeks now and love it, PerfUtil used to be my goto until I found your project. Thanks for all your hard work! I've been trying to evangelize it to the F# community, and I think they're slowly warming to it. Maybe an FSharp.Charting plugin/helper module would make it more enticing....
Andrew Cherry
@kolektiv
@cloudRoutine - you've got it working with f#? i'm trying, but i'm finding it doesn't pick up an f# core. the assemblies and associated .config it generates doesn't have the assembly binding redirects to pick up FSharp.Core - have you found that? or a way round it?
Jared Hester
@cloudRoutine
@kolektiv I didn't run into that issue, got a repo I can inspect?
Jared Hester
@cloudRoutine
@kolektiv this trival example should run without issue
module BDNetFSharp.Program
open BenchmarkDotNet 
open System.Reflection

type ArrayVsList () =
    [<Params (10, 100)>] 
    member val public ColSize = 0 with get, set
    [<Benchmark>]
    member self.GetSquareArray () =
        let value = self.ColSize / 2 in [| for _ in 0 .. self.ColSize -> value |].[value]
    [<Benchmark>]
    member self.GetSquareList () =
        let value = self.ColSize / 2 in [for _ in 0 .. self.ColSize -> value ].[value]

let [<EntryPoint>] main args =
    let competitionSwitch =    
        Assembly.GetExecutingAssembly().GetTypes()
        |> Array.filter (fun t ->
            t.GetMethods (BindingFlags.Instance ||| BindingFlags.Public )
            |> Array.exists (fun m -> m.GetCustomAttributes (typeof<BenchmarkAttribute>, false) <> [||]))
        |> BenchmarkSwitcher
    competitionSwitch.Run args 
    0
Andrey Akinshin
@AndreyAkinshin
@cloudRoutine, I want to add special plugin type for plotting on the next week. It would be cool, if you can create PR with FSharp.Charting support.
Andrey Akinshin
@AndreyAkinshin
analysers.png
@mattwarren, check it out, I added BenchmarkEnvironmentAnalyser: PerfDotNet/BenchmarkDotNet@de01401
Andrew Cherry
@kolektiv
@cloudRoutine - see issue #67 :)
Andrey Akinshin
@AndreyAkinshin
Another project that uses BenchmarkDotNet: http://www.mapperator.net/Benchmark
Andrey Akinshin
@AndreyAkinshin
@mattwarren, I have implemented basic plot support (based on R script) in the refactoring branch.