:D
async void is probably not such a good idea
@bitbonk as of today it's impossible to mark one of the parameters as baseline dotnet/BenchmarkDotNet#881
async setup, cleanup and benchmarks are supported
async void is not supported by design and throws an exception ;)
The docs about the
EtwProfiler seem to be misleading: https://benchmarkdotnet.org/articles/features/etwprofiler.html
If do it as suggested:
private static void Main(string[] args)
=> BenchmarkSwitcher
.FromAssembly(typeof(Program).Assembly)
.Run(
args,
DefaultConfig.Instance
.With(new EtwProfiler()));I get the message
You should select the target benchmark(s). Please, print a number of a benchmark (e.g. '0') or a contained benchmark caption (e.g. 'SerializationBenchmarks')
project name a output is the same name though
the alternative is to use BenchmarkRunner, which does not ask anything but simply runs benchmarks from given type/assembly
@eschneider999 are you customizing the output path? basically BDN searches the folder tree until it finds an .sln file and then it's looking in given folder recursively for a project file of the .dll that defines the benchmarks
Do you have a solution file
the code (in case it's easier to understand it than my description): https://github.com/dotnet/BenchmarkDotNet/blob/2ba30330ec5ec8a9abe284c927277a6fe0fc5a0c/src/BenchmarkDotNet/Toolchains/CsProj/CsProjGenerator.cs#L141
Afternoon All,
I am adding benchmarks to the CommunityCarter/Carter repo. We want to see what perf differences we get from migrating to AspNetCore 3.0 are.
The issue is I am getting a timeout on one of the tests after 5mins. I would love a 2nd pair of eyes to see what I can do differently.
My draft PR is here https://github.com/CarterCommunity/Carter/pull/210/files
Specifically the
I am adding benchmarks to the CommunityCarter/Carter repo. We want to see what perf differences we get from migrating to AspNetCore 3.0 are.
The issue is I am getting a timeout on one of the tests after 5mins. I would love a 2nd pair of eyes to see what I can do differently.
My draft PR is here https://github.com/CarterCommunity/Carter/pull/210/files
Specifically the
Carter.Benchmarks.Harness.UseCarterBenchmark are failing.
Thanks @JohanLarsson
@ritasker I've responded in the PR: https://github.com/CarterCommunity/Carter/pull/210/files#r343083187
I copy benchmark results next to the benchmark code like this: https://github.com/DotNetAnalyzers/WpfAnalyzers/tree/master/WpfAnalyzers.Benchmarks/Benchmarks
And there has been some churn in the copy code due to files changing names
Not a huge issue, more nice to have.
@CyberBotX You can use
InProcessToolchain https://benchmarkdotnet.org/articles/samples/IntroInProcess.html
Good morning. I'm currently trying to use BenchmarkDotNet for performance regression analysis of a library we're authoring. There are basically two things I hope to achieve here:
- Some tests are intended to demonstrate linear scaling of certain API usages (i.e. test doing something with one item or a hundred and not get a 10000× slowdown). (scaling tests)
- Other tests are intended to compare API performance compared with an older version of the library to see whether we've done something very wrong with a change during development. (regression tests)
I've got something working-ish, right now, but encountered a few things:
- I currently create jobs explicitly since testing against different library versions requires custom MSBuild parameters. Creating a config and jobs explicitly seems to be at odds with using attributes to do the same (it appears the attributes get ignored in this case?).
- I do actually need two (maybe even three) different configurations and two subsets of tests (regression tests where the baseline is a number of benchmarks in a completely different job, and scaling tests where the baseline is different for each benchmark). Is it a good idea to put them all into the same project? Should each individual benchmark be its own project instead of its own class? I could probably separate benchmark classes by category well enough, I'm just wondering whether it's a good idea.
- Basically, I'm trying to put this into a CI job that runs daily-ish and use BDN mainly to not worry about statistics. I'd also want to reduce the overhead in creating new tests so that new ones can be added as needed. I do realize that this is probably far from pin-point performance optimization and micro-benchmarks, so perhaps I'm also trying to use the wrong tool here.
and have a dedicated config with per category
sth like
[BenchmarkCategory("SomeCategoryName")]
public class SomeBenchmarks
[BenchmarkCategory("OtherCategoryName")]
public class OtherBenchmarks
IConfig commonSettings = DefaultConfig.Instance; // can be extended using .With
IConfig first = commonSettings
.With(Job.Defeault.WithBuildConfiguration("SOME"))
.With(new AllCategoriesFilter("SomeCategoryName"));
IConfig second = commonSettings
.With(new AllCategoriesFilter("OtherCategoryName"));
BenchmarkRunner.Run(typeof(Program).Assembly, first);
BenchmarkRunner.Run(typeof(Program).Assembly, second);
There seems to be a difference in how projects get built at runtime by BDN and the temporary projects (along with the batch file) that are created. Right now if I'm mistakenly specify a dependency version that doesn't exist in the file system (which should fail the build and test), building and benchmarking works from BDN, but running the batch file individually doesn't work (as expected).
If that can't be solved, I thought I could look at the desired dependency version and the actual assembly version somehow, but couldn't figure out how (or whether) I can access the current job configuration from within a benchmark. I'd find it likely that it isn't supported, though. This would be my pre-benchmark sanity checks, though, which check whether the benchmarks have a hope of yielding useful results.
I fear that the thing I want to do is not possible at the moment
However, I found a (theoretical) workaround by writing some data at
Cleanup to a file and then read it again in the column code