Skip to content

Commit 50c9562

Browse files
authored
Add benchmarking documentation (#50027)
1 parent fbe90bb commit 50c9562

File tree

2 files changed

+100
-0
lines changed

2 files changed

+100
-0
lines changed

docs/Benchmarks.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Benchmarks
2+
3+
This document explains how to benchmark ASP.NET Core changes in the ASP.NET Core perf lab. Currently, only members of the .NET team have the permissions necessary to trigger performance runs on the official infrastructure.
4+
5+
It describes how to trigger benchmarks using the Crank CLI and the Pull-Request bot.
6+
7+
## Benchmarking local changes
8+
9+
This is the recommended method as it is quick, flexible and doesn't require a pre-existing pull-request.
10+
11+
### Install Crank
12+
13+
The Crank dotnet tool is required to start benchmarks remotely on the available performance infrastructure.
14+
15+
Install Crank with the following command:
16+
17+
```console
18+
dotnet tool install Microsoft.Crank.Controller --version "0.2.0-*" --global
19+
```
20+
21+
For more information about Crank please refer to the [Crank GitHub repository](https://github.com/dotnet/crank).
22+
23+
### Choose which benchmark to run
24+
25+
- In the [PowerBI dashboard](https://msit.powerbi.com/groups/b5743765-ec44-4dfc-91df-e32401023530/reports/10265790-7e2e-41d3-9388-86ab72be3fe9/ReportSection30725cd056a647733762?experience=power-bi) select a scenario that you would like to benchmark and the environment to use.
26+
- At the bottom of the page there is table that lists the command lines that were used to execute this benchmark. Pick one.
27+
- Run the command in a shell. This will start the benchmark remotely on the same target machines as the charts.
28+
29+
This command line contains the specific version of the runtimes and SDK that were used, which makes it deterministic. This means that if you want to investigate a regression you can pick two points on a chart and re-run these to confirm the regression. You can also change each version to understand if the regression is coming from ASP.NET or the .NET Runtime.
30+
31+
### Use local changes
32+
33+
Using the same Crank command line you can also test the impact of local changes.
34+
35+
- Build your changes in `release` for the desired architecture/os (if it matters).
36+
- Add `--application.options.outputFiles c:\build\release` and replace the folder by the one containing your changes.
37+
- Run the updated Crank command line
38+
39+
## Benchmarking pull-requests
40+
41+
Use this technique when you want to benchmark a community contributed PR or when you don't have the changes available locally.
42+
43+
### Display all available benchmarks
44+
45+
[Here is an example](https://github.com/dotnet/aspnetcore/pull/50016#issuecomment-1677395856) of a PR that was benchmarked using this bot.
46+
47+
- In the pull-request, add `/benchmark` in the comments.
48+
- A text similar to this will be displayed (it might take up to 10 minutes):
49+
50+
---
51+
Crank Pull Request Bot
52+
53+
`/benchmark <benchmark[,...]> <profile[,...]> <component,[...]> <arguments>`
54+
55+
Benchmarks:
56+
- `plaintext`: TechEmpower Plaintext Scenario - ASP.NET Platform implementation
57+
- `json`: TechEmpower JSON Scenario - ASP.NET Platform implementation
58+
- `fortunes`: TechEmpower Fortunes Scenario - ASP.NET Platform implementation
59+
- `yarp`: YARP - http-http with 10 bytes
60+
- `mvcjsoninput2k`: Sends 2Kb Json Body to an MVC controller
61+
62+
Profiles:
63+
- `aspnet-perf-lin`: INTEL/Linux 12 Cores
64+
- `aspnet-perf-win`: INTEL/Windows 12 Cores
65+
- `aspnet-citrine-lin`: INTEL/Linux 28 Cores
66+
- `aspnet-citrine-win`: INTEL/Windows 28 Cores
67+
- `aspnet-citrine-ampere`: Ampere/Linux 80 Cores
68+
- `aspnet-citrine-amd`: AMD/Linux 48 Cores
69+
70+
Components:
71+
- `kestrel` (Microsoft.AspNetCore.Server.Kestrel `src\Servers\Kestrel\build.cmd`)
72+
- `mvc` (Microsoft.AspNetCore.Mvc `src\Mvc\build.cmd`)
73+
- `routing` (Microsoft.AspNetCore.Routing `src\Http\build.cmd`)
74+
75+
See [prbenchmarks.aspnetcore.config.yml](https://github.com/aspnet/Benchmarks/blob/2bddb9b43ffbc8cfdb4a2c3777d4e5213bc42332/build/prbenchmarks.aspnetcore.config.yml) for more details.
76+
77+
Arguments: any additional arguments to pass through to crank, e.g. `--variable name=value`
78+
79+
---
80+
81+
Where:
82+
- *Benchmark* is the application that will be stressed.
83+
- *Profile* is the environment where the application will run on.
84+
- *Component* is the ASP.NET project under test that will be built and contains your changes.
85+
86+
> [!IMPORTANT]
87+
> Any changes outside of the "component" projects will not be benchmarked.
88+
89+
Create a new comment with the correct benchmarks, profiles, and components to build. For instance:
90+
91+
```console
92+
/benchmark json aspnet-citrine-lin kestrel
93+
```
94+
95+
Once the benchmark has started (can take up to 10 minutes) the bot adds a comment to the PR. Then, when the benchmark is finished the results are added in a separate comment for both your changes and the source branch so you can compare the metrics your care about.
96+
97+
## Other scenarios
98+
99+
For more information please look the [Benchmarks repository](https://github.com/aspnet/benchmarks) where we maintain most of the benchmarks, and the [Crank repository](https://github.com/dotnet/crank).

docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,4 @@ The table below outlines the different docs in this folder and what they are hel
2929
| [Assembly trimming guide](Trimming.md)| Guidance on adding trimming support to an ASP.NET Core assembly | Repo developers who want to help add support for trimming to ASP.NET Core |
3030
| [Adding new Projects to the Repo](AddingNewProjects.md) | Outlines the process of adding new projects (i.e. `.csproj` files) to the repo | Anyone who finds themselves trying to add a new project and including it in the build.
3131
| [Using WebTransport in Kestrel](WebTransport.md) | Outlines how to setup Kestrel to use WebTransport | Anyone looking to support WebTransport |
32+
| [Benchmarking](Benchmarks.md) | Instructions on how to benchmark PRs and local changes | .NET team |

0 commit comments

Comments
 (0)