Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Intro
This PR represents an initial implementation of a new compiler tool to generate a https://docs.codecov.com/docs/codecov-custom-coverage-format coverage report for macro code. Functionally, the tool feels very robust, with a ton of spec coverage. I tested it out on the various macro code within Athena and I'm at the point now where I'm satisfied with how it's working to the point where I feel comfortable opening this PR.
I've created the PR as a draft as it depends on some other PRs, and because it intentionally includes debug code commented out. The intent/goal around this PR is to mainly show off the implementation thus far, and to solicit feedback either in the form of other changes that would make this PR's implementation simpler, or general high level tweaks.
Please go try it out on your own project(s) to see if you can find any problems/issues.
Implementation Overview
At a high level the current implementation adds a new
crystal tool
that when executed enables a newcollect_covered_macro_nodes
property on the program. When enabled, the macro interpreter will collect macro AST nodes via the#collect_covered_node
method. Because a new macro interpreter is used for each expansion, the collected nodes are "flushed" to another array pending processing via the tool after each macro is expanded.The tool itself iterates over each "group" of collected nodes from each expansion, skipping any that do not relate to user code. It then iterates over reach "group", chunking them based on the line number. The chunking ensures that each node can increment that line's count by
1
or0
, not how many nodes are on that single line.From here, we then perform some heuristics to determine if a given line was a hit, partial hit, or a miss. Partial hits are kept track by caching the specific conditionals and if it was a hit/miss on a line so we can know how many paths of the total possibilities we hit on that single line.
Macro
raise
is handled by raising a newSkipMacroCodeCoverageException
that extendsSkipMacroException
. Rescuing this exception allows the tool to run on the code that was collected up to before the exception was raised, while not resulting in an error/running all the other code after it. The exception's message/trace is printed toSTDOUT
while the report is written toSTDERR
. This is esp useful for testing error flows of custom macro logic as it allows the code to assert the proper error was raised, while still allowing to save the coverage report. I.e. the report JSON/exception outputs are not co-mingled. The main benefit of this is preventing the need to run these kind of tests twice, once for the coverage report and once for the actual test assertions.Depends on #15709
Resolves #14880
Open Issues
Will keep a running list of additional things that need handled: