-
Notifications
You must be signed in to change notification settings - Fork 10.4k
Add benchmarking documentation #50027
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# Benchmarks | ||
|
||
This document explains how to benchmark ASP.NET Core changes in the ASP.NET Core perf lab. Currently, only members of the .NET team have the permissions necessary to trigger performance runs on the official infrastructure. | ||
|
||
It describes how to trigger benchmarks using the Crank CLI and the Pull-Request bot. | ||
|
||
danmoseley marked this conversation as resolved.
Show resolved
Hide resolved
|
||
## Benchmarking local changes | ||
|
||
This is the recommended method as it is quick, flexible and doesn't require a pre-existing pull-request. | ||
|
||
### Install Crank | ||
|
||
The Crank dotnet tool is required to start benchmarks remotely on the available performance infrastructure. | ||
sebastienros marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Install Crank with the following command: | ||
|
||
```console | ||
dotnet tool install Microsoft.Crank.Controller --version "0.2.0-*" --global | ||
``` | ||
|
||
For more information about Crank please refer to the [Crank GitHub repository](https://github.com/dotnet/crank). | ||
|
||
### Choose which benchmark to run | ||
|
||
- In the [PowerBI dashboard](https://msit.powerbi.com/groups/b5743765-ec44-4dfc-91df-e32401023530/reports/10265790-7e2e-41d3-9388-86ab72be3fe9/ReportSection30725cd056a647733762?experience=power-bi) select a scenario that you would like to benchmark and the environment to use. | ||
- At the bottom of the page there is table that lists the command lines that were used to execute this benchmark. Pick one. | ||
- Run the command in a shell. This will start the benchmark remotely on the same target machines as the charts. | ||
|
||
This command line contains the specific version of the runtimes and SDK that were used, which makes it deterministic. This means that if you want to investigate a regression you can pick two points on a chart and re-run these to confirm the regression. You can also change each version to understand if the regression is coming from ASP.NET or the .NET Runtime. | ||
|
||
### Use local changes | ||
|
||
Using the same Crank command line you can also test the impact of local changes. | ||
|
||
- Build your changes in `release` for the desired architecture/os (if it matters). | ||
- Add `--application.options.outputFiles c:\build\release` and replace the folder by the one containing your changes. | ||
- Run the updated Crank command line | ||
|
||
## Benchmarking pull-requests | ||
|
||
Use this technique when you want to benchmark a community contributed PR or when you don't have the changes available locally. | ||
sebastienros marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### Display all available benchmarks | ||
|
||
[Here is an example](https://github.com/dotnet/aspnetcore/pull/50016#issuecomment-1677395856) of a PR that was benchmarked using this bot. | ||
|
||
- In the pull-request, add `/benchmark` in the comments. | ||
- A text similar to this will be displayed (it might take up to 10 minutes): | ||
sebastienros marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
--- | ||
Crank Pull Request Bot | ||
|
||
`/benchmark <benchmark[,...]> <profile[,...]> <component,[...]> <arguments>` | ||
|
||
Benchmarks: | ||
- `plaintext`: TechEmpower Plaintext Scenario - ASP.NET Platform implementation | ||
- `json`: TechEmpower JSON Scenario - ASP.NET Platform implementation | ||
- `fortunes`: TechEmpower Fortunes Scenario - ASP.NET Platform implementation | ||
- `yarp`: YARP - http-http with 10 bytes | ||
- `mvcjsoninput2k`: Sends 2Kb Json Body to an MVC controller | ||
|
||
Profiles: | ||
- `aspnet-perf-lin`: INTEL/Linux 12 Cores | ||
- `aspnet-perf-win`: INTEL/Windows 12 Cores | ||
- `aspnet-citrine-lin`: INTEL/Linux 28 Cores | ||
- `aspnet-citrine-win`: INTEL/Windows 28 Cores | ||
- `aspnet-citrine-ampere`: Ampere/Linux 80 Cores | ||
- `aspnet-citrine-amd`: AMD/Linux 48 Cores | ||
|
||
Components: | ||
sebastienros marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- `kestrel` (Microsoft.AspNetCore.Server.Kestrel `src\Servers\Kestrel\build.cmd`) | ||
- `mvc` (Microsoft.AspNetCore.Mvc `src\Mvc\build.cmd`) | ||
- `routing` (Microsoft.AspNetCore.Routing `src\Http\build.cmd`) | ||
|
||
See [prbenchmarks.aspnetcore.config.yml](https://github.com/aspnet/Benchmarks/blob/2bddb9b43ffbc8cfdb4a2c3777d4e5213bc42332/build/prbenchmarks.aspnetcore.config.yml) for more details. | ||
|
||
Arguments: any additional arguments to pass through to crank, e.g. `--variable name=value` | ||
|
||
--- | ||
|
||
Where: | ||
- *Benchmark* is the application that will be stressed. | ||
- *Profile* is the environment where the application will run on. | ||
- *Component* is the ASP.NET project under test that will be built and contains your changes. | ||
sebastienros marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
> [!IMPORTANT] | ||
> Any changes outside of the "component" projects will not be benchmarked. | ||
|
||
Create a new comment with the correct benchmarks, profiles, and components to build. For instance: | ||
|
||
```console | ||
/benchmark json aspnet-citrine-lin kestrel | ||
``` | ||
|
||
Once the benchmark has started (can take up to 10 minutes) the bot adds a comment to the PR. Then, when the benchmark is finished the results are added in a separate comment for both your changes and the source branch so you can compare the metrics your care about. | ||
|
||
## Other scenarios | ||
|
||
For more information please look the [Benchmarks repository](https://github.com/aspnet/benchmarks) where we maintain most of the benchmarks, and the [Crank repository](https://github.com/dotnet/crank). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.