Separate e2e snapshots and enable parallel test suites for percy

[eth3lbert]

This PR addresses #9052 by:

• Separates the e2e percy snapshots from the QUnit tests by prefixing them with e2e in their titles.
• Enables parallel test suites for percy in CI.

---

Changing only the title to separate snapshot sets without enabling parallel test suites is insufficient. This will cause finalized builds to not recognize each other and consider others missing. Therefore, enabling parallel test suites for percy is necessary.

Thanks @Turbo87 for the information and guidance!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.13%. Comparing base (13924a2) to head (f90f31d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9054      +/-   ##
==========================================
+ Coverage   89.11%   89.13%   +0.01%     
==========================================
  Files         280      280              
  Lines       28216    28216              
==========================================
+ Hits        25146    25151       +5     
+ Misses       3070     3065       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Turbo87]

hmmmm....... it looks like the previous snapshots are now shown as "Missing" 😅

I'm wondering if something like https://www.browserstack.com/docs/percy/integrate/parallel-test-suites might help

[eth3lbert]

hmmmm....... it looks like the previous snapshots are now shown as "Missing" 😅

Yes, that's exactly what I have a question about 😅 . I originally thought there would be two sets of snapshots, but I can only see the ones with the "e2e" prefix.

I'm wondering if something like https://www.browserstack.com/docs/percy/integrate/parallel-test-suites might help

I'll take a look to this.

[eth3lbert]

I noticed a deprecation warning:
https://github.com/rust-lang/crates.io/actions/runs/9891816711/job/27323060801#step:2:11

 Warning: The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

I will make a change to address this later.

[eth3lbert]

Here are some notes based on the https://www.browserstack.com/docs/percy/integrate/parallel-test-suites:

• The NONCE (PERCY_PARALLEL_NONCE) needs to be set manually because it could fail in GitHub CI
  • Some CI providers will reuse the same value we are setting the NONCE to on rebuild. For example, GitHub actions reuse the workflows GITHUB_RUN_ID. This will cause the Percy build to fail with an error saying you can’t add new snapshots to an already finalized build (this is because the NONCE matches an existing build and all NONCEs need to be unique).
  • To work around this problem, you will need to create your own NONCE or find a NONCE that your CI provider gives which is unique for each build.
• The total value needs to be set to the number of finalized builds, otherwise the process will get stuck on "receiving".
  • If a build is hanging in the “receiving” state, even after tests have finished, that means the build was never fully finalized.
  • If PERCY_PARALLEL_TOTAL is set to a specific number, make sure there were that many Percy builds finalized in the test run. For example, if PERCY_PARALLEL_TOTAL=4 and there were only 3 builds ran/finalized, Percy will hang in a receiving state. This is because Percy is waiting for 4 finalized builds.
  • If PERCY_PARALLEL_TOTAL is set to-1 (using --parallel), make sure percy build:finalize was called. If it was, make sure PERCY_PARALLEL_NONCE is the same for the tests and the finalize step.

[Turbo87]

👏