These are chat archives for PerfDotNet/BenchmarkDotNet

30th
Jun 2016
nawfalhasan
@nawfalhasan
Jun 30 2016 15:22
Hi benchmark guys
I was looking to benchmark some code and found about BenchmarkDotNet. I have got 3 questions before I attempt.
nawfalhasan
@nawfalhasan
Jun 30 2016 15:43
  1. I see BDN uses Benchmark attribute on methods to benchmark, and that is all! Doesn't that mean some reflection is going behind the scenes? I would assume you're using expression trees behind the scenes to compile the Benchmark methods during runtime to get acceptable numbers. Fair enough, but compiled expressions are always wee bit slower than directly running methods (I would assume compiling expression trees at runtime is not as optimized as the IL generated by compile time compiler in release mode).

    Whatever it is (refelcttion/expression), my question is, if your benchmarking approach is going to run my test methods wee bit slower than it can actually run, doesn't that mean I dont get absolute/raw value (by value I mean time taken to complete the run)? Or is BDN always meant to give relative values (given the said performance bottleneck is true for all the Benchmark attributed methods)?

  2. How do you measure elapsed time? Using Stopwatch class one can get time elapsed, but there is also Process.GetCurrentProcess().TotalProcessorTime which gives the CPU time. Which method does BDN prefer, or we can choose?

  3. I have often found that when running a series of benchmark code I see different results based on the order of execution even when I run warmups (I always ran only one or two warmups instead of a million times, so that could be the reason). That is if I run Test1 and Test2 in order, often Test2 is faster, and if I reverse the order, Test1 is faster. What does BDN do to minimize this effect? Run hundreds of thousands of warmups? Or you randomize the testing order?

I should better see the code and find out myself, but if it's tons of classes it will be overwhelming. Sorry to ask you here :)
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 16:41
  1. No, we don't use reflection, we use code generation/compilation in runrtime.
  2. We use Stopwatch. It will be tricky to produce reproducible results based on TotalProcessorTime.
  3. Yeah, I have a lot of such examples. So, we create a separated process for each benchmark method and run it several times. So, benchmark methods don't affect each other.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:02

@AndreyAkinshin thanks for response, good to know.

Can BDN produce charts? Excel or not. Or any kind of visualization aid?

Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:03
If you have installed R, you can produce a lot of nice plots with help of RPlotExporter.
Look at our documentation: https://github.com/PerfDotNet/BenchmarkDotNet, there a lot of useful information.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:04
Oh cool. So nothing built in from within .NET?
Thanks. Let me give it a look.
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:04
We have some xml-docs, but README contains a lot of explanations, tables, usage examples.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:05
I see.

Regarding point 1, I am just wondering what are the implications of BDN's methodology running at reduced speed (that is if I assume compiled function at runtime is never as performant as compile time compiled function, which is almost always the case). Wouldn't it be better to expose a method that accepts an Action as the test method? Something like:

public double Benchmark(Action action) 
{
    for.............
        action();
}

Benchmark(() => myTestMethod());

This way I assume, the an additional layer of delegate redirection has lesser effect on the quality of ouput than BDN's current approach?

Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:10
We genereate real source code and compile it manually (it's runtime of host process, not the benchmark process) by MSBuild(current stable version 0.9.7) or Roslyn (future version 0.9.8).
And generated code contains some magic based on delegates includes idle method (with the same signature as the target benchmark method) for overhead evaluation.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:14
I see. I was thinking you do Expression.Compile/IL.Emit sort of thing. Wow that seems so advanced then. Good to know.
And github readme is amazing, so much info.
And looks like it supports .NET Core as well, I am sold! Thanks guys!
nawfalhasan
@nawfalhasan
Jun 30 2016 17:26
For plots, just got to install R, that's it?
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:27

And looks like it supports .NET Core as well, I am sold! Thanks guys!

Yep. Only RC1/RC2 for now, but we are planning to include RTM support in 0.9.8.

For plots, just got to install R, that's it?

And use a config which include RPlot export like the following:

public class Config : ManualConfig
{
    public Config()
    {
        Add(CsvMeasurementsExporter.Default);
        Add(RPlotExporter.Default);
    }
}
nawfalhasan
@nawfalhasan
Jun 30 2016 17:28
And have R installed, that's all?
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:28
Yep.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:29
Cool. Why didnt I find this earlier :P Any rough projection when will 0.9.8 be out?
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:29
RPlotExporter will generate a R script and automatically run it. But you can also add some additional logic and rerun script manually.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:30
Hmm. Why chose R for this? .NET didnt have any libraries around?
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:30
I have some troubles with new Roslyn toolchin for benchmark generation. As soon as I solved it, it will be released. Hopefully, soon. =)
nawfalhasan
@nawfalhasan
Jun 30 2016 17:31
Nice.
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:31
I didn't found a nice xplat library with DNX/CoreCLR support that produces nice plots.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:32
I see.
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:32
Also I just know R well, it was a quick easy way to add plots.
We have a plugin system, so, we will add other plot engines in the future.
You can even write you own plugin.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:33
:+1:
I see a lot of configuration styles in readme. Is there a way to have it in app.config itself?
I mean all BDN settings in a config file rather than in code?
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:36
No.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:36
Hmm.
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 17:39
You can specify configs via command line.
nawfalhasan
@nawfalhasan
Jun 30 2016 17:45
How many warmups do BDN do by default? I kow I can specify in config, but if I dont?
Also what is TargetCount?
Andrey Akinshin
@AndreyAkinshin
Jun 30 2016 19:37
We have some tricky logic which tries to adjust each parameter automatically based on a set of pilot measurements.
nawfalhasan
@nawfalhasan
Jun 30 2016 22:05
Hmm.