These are chat archives for PerfDotNet/BenchmarkDotNet

1st
Feb 2016
Matt Warren
@mattwarren
Feb 01 2016 16:36
@AndreyAkinshin just FYI with the latest code, I managed to get this impressive run, where Result and MainTarget iterations went pretty high, not sure it it's expected or not?
MainTarget 314: 16 op, 247928584.73 ns, 15.4955 ms/op
MainTarget 315: 16 op, 247917561.51 ns, 15.4948 ms/op
...
Result 314: 16 op, 247928204.62 ns, 15.4955 ms/op
Result 315: 16 op, 247917181.4 ns, 15.4948 ms/op
Andrey Akinshin
@AndreyAkinshin
Feb 01 2016 21:24

It is the proof that new system works well! Let's see at the middle of benchmark:

MainTarget 192: 32 op, 497058700.55 ns, 15.5331 ms/op
MainTarget 193: 32 op, 319215754.84 ns, 9.9755 ms/op

Here we have a really big variance. We just can't measure the method based on 10–20 iterations. So, BenchmarkDotNet try to increase total iteration amount to get better results

IntroBasic_SleepWithDescription
Mean = 14.0604 ms, StdError = 0.1359 ms (0.97%); N = 272, StdDev = 2.2406 ms
Min = 9.9751 ms, Q1 = 12.4934 ms, Median = 15.5459 ms, Q3 = 15.5658 ms, Max = 15.9710 ms
IQR = 3.0723 ms, LowerFence = 7.8849 ms, UpperFence = 20.1743 ms
ConfidenceInterval = [13.7941 ms; 14.3267 ms] (CI 95%)

IntroBasic_Sleep
Mean = 13.9145 ms, StdError = 0.1351 ms (0.97%); N = 325, StdDev = 2.4354 ms
Min = 9.9413 ms, Q1 = 10.1569 ms, Median = 15.4953 ms, Q3 = 15.5094 ms, Max = 15.9920 ms
IQR = 5.3525 ms, LowerFence = 2.1282 ms, UpperFence = 23.5380 ms
ConfidenceInterval = [13.6497 ms; 14.1792 ms] (CI 95%)

Your measurements is very interesting. Do you have installed R? I recommend you to try our new plot system, we have new cool density plots in "Big API refactoring, Part 7 ".
BTW, I will return StdDev instead of StdError in the summary table.