Skip to content

Summary table #1335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Summary table #1335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
463112c
Create summary table
rylev May 20, 2022
df53860
Move table to template and add filtered table
rylev May 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion site/src/load.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ impl SiteCtxt {
.map(|bench| {
(
bench.name.as_str().into(),
Category::from_db_representation(&bench.category).unwrap(),
Category::from_db_representation(&bench.category).unwrap_or(Category::Primary),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was getting panics locally without this. Let me know if this should be removed or if we should unwrap into Category::Secondary.

)
})
.collect()
Expand Down
232 changes: 144 additions & 88 deletions site/static/compare.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@
margin: 0;
padding: 0;
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the formatting changes being included in here. Let me know if this is too much a bother, and I can try to remove them.

.section-list-wrapper {
flex-direction: column;
}
Expand All @@ -129,11 +130,13 @@
align-items: center;
width: 80%;
}

.section-list-wrapper {
flex-direction: row;
}
}
.states-list > li {

.states-list>li {
margin-right: 15px;
}

Expand Down Expand Up @@ -227,10 +230,12 @@
width: 25%;
min-width: 50px;
}

.benches td {
text-align: center;
width: 25%;
}

.benches td.numeric {
text-align: right;
}
Expand All @@ -254,37 +259,45 @@
flex-direction: column;
margin-bottom: 10px;
}

.summary {
display: flex;
}

.summary-values {
display: flex;
flex-direction: column;
}

@media (min-width: 650px) {
.summary-container {
flex-direction: row;
margin-bottom: 0;
align-items: center;
}
.summary-container > span {

.summary-container>span {
text-align: right;
width: 20%;
}

.summary-values {
width: 100%;
flex-direction: row;
justify-content: flex-end;
align-items: center;
}

.summary {
width: 15%;
align-items: center;
}

.summary-wide {
width: 20%;
}
}

.category-title {
font-weight: bold;
font-size: 1.2em;
Expand Down Expand Up @@ -404,8 +417,7 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
</li>
<li>
<label>
<input type="checkbox" id="profile-opt"
v-model="filter.profile.opt" />
<input type="checkbox" id="profile-opt" v-model="filter.profile.opt" />
<span class="cache-label">opt</span>
</label>
<div class="tooltip">?
Expand All @@ -416,8 +428,7 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
</li>
<li>
<label>
<input type="checkbox" id="profile-doc"
v-model="filter.profile.doc" />
<input type="checkbox" id="profile-doc" v-model="filter.profile.doc" />
<span class="cache-label">doc</span>
</label>
<div class="tooltip">?
Expand Down Expand Up @@ -532,10 +543,11 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
<div class="section-heading"><span>Show non-relevant results</span>
<span class="tooltip">?
<span class="tooltiptext">
Whether to show test case results that are not relevant (i.e., not significant or
have a large enough magnitude). You can see
<a href="https://github.com/rust-lang/rustc-perf/blob/master/docs/comparison-analysis.md#how-is-relevance-of-a-test-run-summary-determined">
here</a> how relevance is calculated.
Whether to show test case results that are not relevant (i.e., not significant or
have a large enough magnitude). You can see
<a
href="https://github.com/rust-lang/rustc-perf/blob/master/docs/comparison-analysis.md#how-is-relevance-of-a-test-run-summary-determined">
here</a> how relevance is calculated.
</span>
</span>
</div>
Expand Down Expand Up @@ -565,39 +577,10 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
</span>
</span>
</div>
<div v-for="summaryPair in Object.entries(summary)" class="summary-container">
<span style="font-weight: bold; margin-left: 5px; text-transform: capitalize;">{{
summaryPair[0] }}:</span>
<div class="summary-values">
<span class="summary summary-wide positive">
{{summaryPair[1].regressions.toString().padStart(3, "&nbsp;")}}
<svg style="width:18px;height:18px" viewBox="0 0 24 24">
<path
d="M16,6L18.29,8.29L13.41,13.17L9.41,9.17L2,16.59L3.41,18L9.41,12L13.41,16L19.71,9.71L22,12V6H16Z">
</path>
</svg>
&nbsp;(+{{(summaryPair[1].regressions_avg).toFixed(2)}}%)
</span>
<span class="summary">
{{summaryPair[1].unchanged.toString().padStart(3, "&nbsp;")}}
<svg style="width:18px;height:18px" viewBox="0 0 24 24">
<path d="M22,12L18,8V11H3V13H18V16L22,12Z"></path>
</svg>
</span>
<span class="summary summary-wide negative">
{{summaryPair[1].improvements.toString().padStart(3, "&nbsp;")}}
<svg style="width:18px;height:18px" viewBox="0 0 24 24">
<path
d="M16,18L18.29,15.71L13.41,10.83L9.41,14.83L2,7.41L3.41,6L9.41,12L13.41,8L19.71,14.29L22,12V18H16Z">
</path>
</svg>
&nbsp;({{(summaryPair[1].improvements_avg).toFixed(2)}}%)
</span>
<span class="summary" v-bind:class="percentClass(summaryPair[1].average)">
&nbsp;{{ signIfPositive(summaryPair[1].average) }}{{ (summaryPair[1].average).toFixed(2) }}%
</span>
</div>
</div>
<p>All:</p>
<summary-table :summary="summary.all"></summary-table>
<p>Filtered:</p>
<summary-table :summary="summary.filtered"></summary-table>
</div>
<div v-if="data.new_errors.length">
<p><b>Newly broken benchmarks</b>:</p>
Expand All @@ -608,24 +591,12 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
<hr />
</div>

<test-cases-table
title="Primary"
:cases="testCases.filter(c => c.category === 'primary')"
:show-raw-data="showRawData"
:commit-a="data.a"
:commit-b="data.b"
:stat="stat"
:before="before"
<test-cases-table title="Primary" :cases="testCases.filter(c => c.category === 'primary')"
:show-raw-data="showRawData" :commit-a="data.a" :commit-b="data.b" :stat="stat" :before="before"
:after="after"></test-cases-table>
<hr />
<test-cases-table
title="Secondary"
:cases="testCases.filter(c => c.category === 'secondary')"
:show-raw-data="showRawData"
:commit-a="data.a"
:commit-b="data.b"
:stat="stat"
:before="before"
<test-cases-table title="Secondary" :cases="testCases.filter(c => c.category === 'secondary')"
:show-raw-data="showRawData" :commit-a="data.a" :commit-b="data.b" :stat="stat" :before="before"
:after="after"></test-cases-table>
<br />
<hr />
Expand Down Expand Up @@ -679,7 +650,7 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
}

const app = Vue.createApp({
mounted() {
mounted() {
const app = this;
loadState(state => makeData(state, app));

Expand Down Expand Up @@ -782,7 +753,7 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
scenarioFilter(testCase.scenario) &&
categoryFilter(testCase.category) &&
relevanceFilter &&
nameFilter
nameFilter
);
}

Expand Down Expand Up @@ -888,29 +859,72 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
sum[testCaseString(next)] = true;
return sum;
}, {});
const newCount = {
regressions: 0,
regressions_avg: 0,
improvements: 0,
improvements_avg: 0,
unchanged: 0,
average: 0
};
const benchmarks = {};
for (const benchmark of this.data.benchmark_data) {
benchmarks[benchmark.name] = benchmark.category;
}

const addDatum = (result, datum, percent) => {
if (percent > 0 && datum.is_relevant) {
result.regressions += 1;
result.regressions_avg += percent;
} else if (percent < 0 && datum.is_relevant) {
result.improvements += 1;
result.improvements_avg += percent;
const originalResult = result;
if (!datum.is_relevant) {
return;
}
let category = benchmarks[datum.benchmark];
if (category === "primary") {
result = result.primary;
} else {
result.unchanged += 1;
result = result.secondary;
}
if (percent > 0) {
result = result.regressions;
} else if (percent < 0) {
result = result.improvements;
}

let compute = (result) => {
result.count += 1;
// Incremental average: https://math.stackexchange.com/questions/106700/incremental-averaging
result.mean += (percent - result.mean) / result.count;
result.max = Math.max(result.max, percent);
}
compute(result)
if (category == "primary") {
compute(originalResult.primary.all)
}
result.average += percent;
};

let result = { all: { ...newCount }, filtered: { ...newCount } }
const newCount = {
primary: {
regressions: {
count: 0,
mean: 0,
max: 0,
},
improvements: {
count: 0,
mean: 0,
max: 0,
},
all: {
count: 0,
mean: 0,
max: 0,
}
},
secondary: {
regressions: {
count: 0,
mean: 0,
max: 0,
},
improvements: {
count: 0,
mean: 0,
max: 0,
},
}
};
let result = { all: { ...JSON.parse(JSON.stringify(newCount)) }, filtered: { ...newCount } }
for (let d of this.data.comparisons) {
const testCase = testCaseString(d)
const datumA = d.statistics[0];
Expand All @@ -922,14 +936,6 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
}
}

const computeAvg = (result) => {
result.improvements_avg /= Math.max(result.improvements, 1);
result.regressions_avg /= Math.max(result.regressions, 1);
result.average /= Math.max(result.regressions + result.improvements + result.unchanged, 1);
};
computeAvg(result.all);
computeAvg(result.filtered);

return result;

},
Expand Down Expand Up @@ -1087,7 +1093,57 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
</tbody>
</table>
</div>
`});
`
});

app.component('summary-table', {
props: ['summary'],
methods: {
formatSummaryFigure(figure) {
if (!figure) {
return "N/A";
}
return figure.toFixed(2) + "%";
}
},
template: `
<table>
<tr>
<th></th>
<th>Regressions (primary)</th>
<th>Regressions (secondary)</th>
<th>Improvements (primary)</th>
<th>Improvements (secondary)</th>
<th>All (primary)</th>
</tr>
<tr>
<td>count</td>
<td>{{ summary.primary.regressions.count }}</td>
<td>{{ summary.secondary.regressions.count }}</td>
<td>{{ summary.primary.improvements.count }}</td>
<td>{{ summary.secondary.improvements.count }}</td>
<td>{{ summary.primary.all.count }}</td>
</tr>
<tr>
<td>mean</td>
<td>{{ formatSummaryFigure(summary.primary.regressions.mean) }}</td>
<td>{{ formatSummaryFigure(summary.secondary.regressions.mean) }}</td>
<td>{{ formatSummaryFigure(summary.primary.improvements.mean) }}</td>
<td>{{ formatSummaryFigure(summary.secondary.improvements.mean) }}</td>
<td>{{ formatSummaryFigure(summary.primary.all.mean) }}</td>
</tr>
<tr>
<td>max</td>
<td>{{ formatSummaryFigure(summary.primary.regressions.max) }}</td>
<td>{{ formatSummaryFigure(summary.secondary.regressions.max) }}</td>
<td>{{ formatSummaryFigure(summary.primary.improvements.max) }}</td>
<td>{{ formatSummaryFigure(summary.secondary.improvements.max) }}</td>
<td>{{ formatSummaryFigure(summary.primary.all.max) }}</td>
</tr>
</table>
`
});

app.mixin({
methods: {
percentClass(pct) {
Expand Down Expand Up @@ -1161,4 +1217,4 @@ <h2>Comparing <span id="stat-header">{{stat}}</span> between <span id="before">{
</script>
</body>

</html>
</html>