Make run-tests GitHub Actions aware

[TimWolla]

No description provided.

[iluuu1994]

The colorization of the state already works, so I'm wondering how much value this is adding 🙂

[TimWolla]

The colorization of the state already works, so I'm wondering how much value this is adding slightly_smiling_face

Hey, this is a Draft, stop looking at it.

But as you already did: I'm annoyed by needing to search around in the log to find which tests exactly failed. With this I'm attempting to directly list the test names in the checks tab. It should work once one of the jobs finishes. A similar example is here: https://github.com/haproxy/haproxy/actions/runs/2690570393

An alternative to the change is adding a problem matcher: https://github.com/actions/toolkit/blob/main/docs/problem-matchers.md which does the same thing. But that one might be a little messy with needing to ignore the color codes during matching.

Having native support would also allow us to leverage “markdown job summaries” which can be used to do something like this: https://github.com/TimWolla/haproxy-auth-request/actions/runs/2725531285. Of course it would need to be discussed what information would be useful to have in that markdown summary.

[TimWolla]

If we ignore the fact that I accidentally marked the skipped tests as warnings, it works more or less like I wanted it to work:

https://github.com/php/php-src/actions/runs/2741510828

[cmb69]

In my opinion, seeing which test failed is not really helpful; instead the diff(s) would likely be.

[TimWolla]

In my opinion, seeing which test failed is not really helpful; instead the diff(s) would likely be.

That should be possible with the https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/. The regular "error" markers I've used here do not support multi-line messages.

[TimWolla]

If we ignore the fact that I accidentally marked the skipped tests as warnings, it works more or less like I wanted it to work:

Another thing this PR does as of now: The test files will also be "annotated" in the "Files Changed" tab. Either as an unchanged file with failures, or inline in the diff if the test itself is also modified.

[iluuu1994]

Hey, this is a Draft, stop looking at it.

Absolutely, my bad.

With this I'm attempting to directly list the test names in the checks tab.

Ah , gotcha.

I'm annoyed by needing to search around in the log to find which tests exactly failed. ... That should be possible with the https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/.

That looks interesting. We could probably also achieve that if we didn't log skipped tests in the CI. (You could still save a few clicks by having a summary right there without clicking into the job)

Another thing this PR does as of now: The test files will also be "annotated" in the "Files Changed" tab. Either as an unchanged file with failures, or inline in the diff if the test itself is also modified.

That sounds useful (as long as skipped/XFAIL tests don't all show up). It's probably hard to show a correct diff though. In it's current form it's verbose and not too meaningful.

• A huge block is generated for each job
• The test can't be expanded, so the only really useful information is the test path

(Sorry I know this is a draft, these are just thoughts not criticism)

[TimWolla]

as long as skipped/XFAIL tests don't all show up

Yeah, that's a bug. I've taken the switch() from the colorization and didn't realize the default: was not just warnings, but also SKIP.

It's probably hard to show a correct diff though.

It might actually be possible thanks to @staabm's comment. I'll look into that, but for now I've pushed a first change to correctly ignore SKIPped tests, which should already make the output much cleaner.

[TimWolla]

For posterity some screenshots:

Job Summary of https://github.com/php/php-src/actions/runs/2742807430
👇

Files Changed
👇

[krakjoe]

echo #9255 (comment)

[iluuu1994]

Let's close this for now as there is no clear consensus on how to move forward. I'm not against improving GitHub integration at all, but I also understand that run-tests.php should be somewhat platform agnostic. Maybe we can move the given logic into an adapter that we can provide just for php-src?

[TimWolla]

Let's close this for now as there is no clear consensus on how to move forward.

Yeah, that's fair.

---

I don't plan on pursuing this further for now. With the greatly slimmed-down output of run-tests in CI, viewing the log files is manageable.