PHPLIB-582: Create JSON test result file for evergreen #787
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmikola
reviewed
Sep 18, 2020
| use function round; | ||
|
|
||
| // phpcs:disable SlevomatCodingStandard.Exceptions.ReferenceThrowableOnly.ReferencedGeneralException | ||
| class EvergreenLogListener extends Printer implements TestListener |
There was a problem hiding this comment.
Since we already use symfony/phpunit-bridge, I noticed it has an alias for a SymfonyTestsListener interface. I think you could leverage that instead of implementing PHPUnit's interface directly and preserve our ability to test with PHPUnit 8.x.
There was a problem hiding this comment.
Unfortunately, that class is not a generic implementation for test listeners, but rather a specific one to support the use-case of the PHPUnit bridge.
There was a problem hiding this comment.
Ah, I just poked my head into SymfonyTestsListenerTrait.
jmikola
approved these changes
Sep 21, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PHPLIB-582
This ticket unfortunately is incomplete due to a few issues with the build system.
Reporting test results using
attach.resultsdoes not allow us to easily add logs (e.g. information about assertion failures, traces, test output, etc.) to the test results. This is possible usingattach.xunit_results, but I could not get that command to read the results files written by PHPUnit. For one, PHPUnit reports nested test suites (a feature of JUnit), which the parser in evergreen does not understand. This should be fixed by EVG-12951.While I had a temporary listener that worked around this by flattening the file (and to report a single test suite containing all tests), this still didn't yield any results. This went as far as Evergreen happily accepting a non-existent or invalid xUnit result file. So, for the time being, we report test results using the JSON format. This will at least cause them to show up in a "Tests" tab under individual tasks, allowing us to see which test failed without consulting the log. Once the problems mentioned above have been solved, we can revisit this and add more detail to the test reporting.
Note that due to how PHPUnit approaches its PHP version support, we cannot support both PHPUnit 6 and PHPUnit 8 anymore. While we could add return type hints in any class implementing the listener interface, we can't do so until we drop PHP 7.0 as the listener interface requires a
voidreturn type which was only introduced in 7.1. The easiest way to fix this is to drop PHPUnit 8 and rely on 6 only, which will work up to PHP 7.4. We can then revisit this once we start removing our custom listeners, drop support for PHP 7.0, or add support for PHP 8.