Skip to content

Commit 3f179f3

Browse files
authored
Increase # of independent samples for changes.
Multimodal benchmarks with significant delta between the modes can report false performance changes when we gather too few independent samples. This increases the minimal number of independent samples from 5 to 10. Fix for https://bugs.swift.org/browse/SR-9907
1 parent 498b722 commit 3f179f3

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

benchmark/scripts/run_smoke_bench

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -185,7 +185,7 @@ def test_performance(opt_level, old_dir, new_dir, threshold, num_samples,
185185
tests = TestComparator(results[0], results[1], threshold)
186186
changed = tests.decreased + tests.increased
187187

188-
while len(changed) > 0 and unchanged_length_count < 5:
188+
while len(changed) > 0 and unchanged_length_count < 10:
189189
i += 1
190190
if VERBOSE:
191191
log(' test again: ' + str([test.name for test in changed]))

0 commit comments

Comments
 (0)