Skip to content

[CI] Tune nightly benchmarking job for better reliability #17122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
Mar 13, 2025

Conversation

ianayl
Copy link
Contributor

@ianayl ianayl commented Feb 21, 2025

This PR tunes the nightly benchmarking job to produce more consistent results:

  • Lowers the tolerance threshold of benchmarking results accepted from 50% to 8%
    • Nightly was flaking before even with a 50% tolerance threshold
  • Raises the iterations to 5000
    • Using 10,000 iterations did not result in significantly more stable performance, although this may change as we obtain more data
    • However, the PVC benchmarking job in the overall nightly workflow now takes about ~47 minutes, whereas before the PVC benchmarking job took ~14 minutes
      • This should not have major impact on execution time however, considering the E2E tests take ~42 minutes: Since both these jobs run in parallel on different machines, the theoretical effect on the overall workflow should only be about 5 minutes, although this would depend on whether or not machines are able to be scheduled in time.
  • Changes the benchmarking workflows in sycl-nightly.yml to use the tuned PERF_PVC runner
    • Untuned machines are exhibiting large variations when running compute-benchmarks (20-25%, up to 50% in the worst case scenario): These are unacceptable variations and not particularly useful.
  • Disables nightly benchmarking on gen12:
    • Gen12 machines are currently untuned. Similar to PVC machines, these results are not accurate and not worth serious nightly benchmarking.
  • Adds guards for benchmarking jobs to prevent benchmark runs in forks [CI] Add CI workflow to run compute-benchmarks on incoming syclos PRs #14454 (comment)

@ianayl ianayl changed the title Switch to PVC_PERF runner for nightly benchmarking [CI] Switch to PVC_PERF runner for nightly benchmarking Feb 21, 2025
Copy link
Contributor

@sarnex sarnex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comments

- name: Compute CPU core range to run benchmarks on
shell: bash
run: |
# Taken from ur-benchmark-reusable.yml:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we make it a shared action in devops/actions if its called by UR CI also?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we combine both CI's there should only be 1 invocation of this, but that's when the dust all settles

Co-authored-by: Nick Sarnie <[email protected]>
@ianayl
Copy link
Contributor Author

ianayl commented Mar 13, 2025

@intel/llvm-gatekeepers PR ready for merge, thanks!

@sarnex sarnex merged commit e38db3a into sycl Mar 13, 2025
33 checks passed
@bader bader deleted the ianayl/tune-benchmarks branch March 18, 2025 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants