Extensibility is a really good idea. I want to implement a system of plugins such that our users can form a desired plugin set from the core, configure it, or write own plugins. My nearest problem is the result export logic, next I want to implement a warning system (messages like
Hey, your StdDev is too big, you should pay attention to it) and toolchain system (custom Generator/Builder/Executor configurations). What do you think?
But how to go about it?
I toyed with creating an issue for extensibility, but I think I'll leave it to you fellas. :)
It would be great to have a sample showing off this plugin stuff so we can see the API you intend and then follow that through to check out implementation.
Also I have implemented new kind of plugins: analysers, see PerfDotNet/BenchmarkDotNet@527df65
@smudge202 @mattwarren what do you think?
@smudge202 @mattwarren what do you think?
Guys, now we have big amount of upcoming features, idea and plans, discussions. What do you think about a wiki page with a roadmap?
Yeah, we need to put this somewhere, even if it's only a GitHub discussion. For instnace I've got several ideas for new Diagnostics that can go on the list
@mattwarren, thanks!
I created a wiki page: https://github.com/PerfDotNet/BenchmarkDotNet/wiki Feel free to make any changes.
I created a wiki page: https://github.com/PerfDotNet/BenchmarkDotNet/wiki Feel free to make any changes.
@Metallium, right now you can reduce amount of processes (the first parameter of
BenchmarkTask) , amount of warmup and target iterations (the two last parameters). E.g., processCount=1, warmupIterationCount=3, targetIterationCount=5. I am trying to implement smart logic that will be adjust these parameters based on you benchmark: if you really don't need super precision, benchmarking will perform quickly. Hopefully, new logic will be published in December.
@smudge202, cool. Waiting for your PR.
[BenchmarkTask(mode: BenchmarkMode.SingleRun, processCount: 1, warmupIterationCount: 1, targetIterationCount: 1)] i.e. "SingleRun"
For example, allowing @Metallium to create a
QuickBenchmarkTaskAttribute with settings as you've described.
public class QuickBenchmarkTaskAttribute : BenchmarkTaskAttribute
{
public QuickBenchmarkTaskAttribute()
: base(mode: BenchmarkMode.SingleRun, processCount: 1, warmupIterationCount: 1, targetIterationCount: 1)
{ }
}
Because the
GetCustomAttributes call linked above states false for allow inheritance
https://msdn.microsoft.com/en-us/library/ms130871(v=vs.110).aspx <= we need to set
inherit: true to discover derived attributes I believe
I believe the code that's there will still work
Installing VS2015 U1 so can't check in VS, and too lazy to update VSCode... sorry. :)
I'm going to create an associated bug. Either
BenchmarkAttribute should be sealed or the GetCustomAttributes call should allow inheritance. The combination we have right now I consider to be a bug. Obviously, changing the call instead of sealing the attribute would be a better fix.